---CRS1 ( THIS IS UP AND RUNNING FINE ) 
oracle@crs1 ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    crs1
ora....S1.lsnr application    ONLINE    ONLINE    crs1
ora.crs1.gsd   application    ONLINE    ONLINE    crs1
ora.crs1.ons   application    ONLINE    ONLINE    crs1
ora.crs1.vip   application    ONLINE    ONLINE    crs1
ora....SM2.asm application    ONLINE    ONLINE    crs2
ora....S2.lsnr application    ONLINE    ONLINE    crs2
ora.crs2.gsd   application    ONLINE    ONLINE    crs2
ora.crs2.ons   application    ONLINE    ONLINE    crs2
ora.crs2.vip   application    ONLINE    ONLINE    crs2
ora....b1.inst application    ONLINE    ONLINE    crs1
ora....b2.inst application    ONLINE    ONLINE    crs2
ora.crsdb.db   application    ONLINE    ONLINE    crs1

oracle@crs1 admin]$ ps -ef | grep pmon
oracle   10043     1  0 May17 ?        00:00:01 asm_pmon_+ASM1
oracle   10867     1  0 May17 ?        00:00:01 ora_pmon_crsdb1
oracle    5339 26044  0 00:13 pts/1    00:00:00 grep pmon
[oracle@crs1 admin]$ ps -ef | grep -i list
root      7417  7379  0 May17 ?        00:00:16 /usr/X11R6/bin/X :0 -audit 0 -auth /var/gdm/:0.Xauth -nolisten tcp vt7
oracle    9798     1  0 May17 ?        00:00:00 /u01/app/oracle/product/10.2.0/asm_1/bin/tnslsnr LISTENER_CRS1 -inherit
oracle    5396 26044  0 00:13 pts/1    00:00:00 grep -i list


---CRS2


oracle@crs2 bdump]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    crs1
ora....S1.lsnr application    ONLINE    ONLINE    crs1
ora.crs1.gsd   application    ONLINE    ONLINE    crs1
ora.crs1.ons   application    ONLINE    ONLINE    crs1
ora.crs1.vip   application    ONLINE    ONLINE    crs1
ora....SM2.asm application    ONLINE    OFFLINE
ora....S2.lsnr application    ONLINE    OFFLINE
ora.crs2.gsd   application    ONLINE    ONLINE    crs2
ora.crs2.ons   application    ONLINE    ONLINE    crs2
ora.crs2.vip   application    ONLINE    OFFLINE
ora....b1.inst application    ONLINE    ONLINE    crs1
ora....b2.inst application    ONLINE    OFFLINE
ora.crsdb.db   application    ONLINE    ONLINE    crs1


--rdbms instance alert log ( CRS2 ) 

Completed: ALTER DATABASE OPEN
Fri May 17 23:11:52 2013
Shutting down instance (abort)
License high water mark = 8
Instance terminated by USER, pid = 16368
Sat May 18 00:08:05 2013
Starting ORACLE instance (normal)
Cannot determine all dependent dynamic libraries for /proc/self/exe
Unable to find dynamic library libocr10.so in search paths
RPATH = /ade/aime1_build2101/oracle/has/lib/:/ade/aime1_build2101/oracle/lib/:/ade/aime1_build2101/oracle/has/lib/:
LD_LIBRARY_PATH is not set!
The default library directories are /lib and /usr/lib
Unable to find dynamic library libocrb10.so in search paths
Unable to find dynamic library libocrutl10.so in search paths
Unable to find dynamic library libocrutl10.so in search paths
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 192.168.2.0 configured from OCR for use as  a public interface
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_10 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  processes                = 150
  __shared_pool_size       = 96468992
  __large_pool_size        = 4194304
  __java_pool_size         = 4194304
  __streams_pool_size      = 0
  spfile                   = +DATA/crsdb/spfilecrsdb.ora
  sga_target               = 285212672
  control_files            = +DATA/crsdb/controlfile/current.260.815657249, +FRA/crsdb/controlfile/current.256.815657249
  db_block_size            = 8192
  __db_cache_size          = 176160768
  compatible               = 10.2.0.1.0
  db_file_multiblock_read_count= 16
  cluster_database         = TRUE
  cluster_database_instances= 2
  db_create_file_dest      = +DATA
  db_recovery_file_dest    = +FRA
  db_recovery_file_dest_size= 4294967296
  thread                   = 2
  instance_number          = 2
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS2
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (PROTOCOL=TCP) (SERVICE=crsdbXDB)
  remote_listener          = LISTENERS_CRSDB
  job_queue_processes      = 10
  background_dump_dest     = /u01/app/oracle/admin/crsdb/bdump
  user_dump_dest           = /u01/app/oracle/admin/crsdb/udump
  core_dump_dest           = /u01/app/oracle/admin/crsdb/cdump
  audit_file_dest          = /u01/app/oracle/admin/crsdb/adump
  db_name                  = crsdb
  open_cursors             = 300
  pga_aggregate_target     = 94371840
Cluster communication is configured to use the following interface(s) for this instance
  10.10.10.32
Sat May 18 00:08:06 2013
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=26664
DIAG started with pid=3, OS id=26666
LMD0 started with pid=6, OS id=26679
LMON started with pid=5, OS id=26677
LMS0 started with pid=7, OS id=26681
PSP0 started with pid=4, OS id=26675
MMAN started with pid=8, OS id=26691
DBW0 started with pid=9, OS id=26693
CKPT started with pid=11, OS id=26697
LGWR started with pid=10, OS id=26695
SMON started with pid=12, OS id=26699
RECO started with pid=13, OS id=26701
MMNL started with pid=16, OS id=26707
Sat May 18 00:08:06 2013
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMON started with pid=15, OS id=26705
Sat May 18 00:08:07 2013
starting up 1 shared server(s) ...
CJQ0 started with pid=14, OS id=26703
Sat May 18 00:08:07 2013
lmon registered with NM - instance id 2 (internal mem no 1)
Sat May 18 00:08:08 2013
Reconfiguration started (old inc 0, new inc 16)
List of nodes:
 0 1
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
 Communication channels reestablished
 * domain 0 valid = 1 according to instance 0
Sat May 18 00:08:09 2013
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Sat May 18 00:08:09 2013
 LMS 0: 0 GCS shadows cancelled, 0 closed
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
Sat May 18 00:08:09 2013
 LMS 0: 0 GCS shadows traversed, 0 replayed
Sat May 18 00:08:09 2013
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=19, OS id=26789
Sat May 18 00:08:10 2013
ALTER DATABASE   MOUNT
Sat May 18 00:08:10 2013
Starting background process ASMB
ASMB started with pid=21, OS id=26793
Starting background process RBAL
RBAL started with pid=22, OS id=26798
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Sat May 18 00:08:14 2013
SUCCESS: diskgroup DATA was mounted
SUCCESS: diskgroup FRA was mounted
Sat May 18 00:08:19 2013
Setting recovery target incarnation to 2
Sat May 18 00:08:19 2013
Successful mount of redo thread 2, with mount id 644929121
Sat May 18 00:08:19 2013
Allocated 3981204 bytes in shared pool for flashback generation buffer
Starting background process RVWR
RVWR started with pid=25, OS id=27072
Sat May 18 00:08:19 2013
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE   MOUNT
Sat May 18 00:08:21 2013
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Sat May 18 00:08:22 2013
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=26, OS id=27160
Sat May 18 00:08:22 2013
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=27, OS id=27164
Sat May 18 00:08:22 2013
Thread 2 opened at log sequence 4
  Current log# 4 seq# 4 mem# 0: +DATA/crsdb/onlinelog/group_4.267.815657449
  Current log# 4 seq# 4 mem# 1: +FRA/crsdb/onlinelog/group_4.260.815657449
Sat May 18 00:08:22 2013
ARC1: STARTING ARCH PROCESSES
Sat May 18 00:08:22 2013
Successful open of redo thread 2
Sat May 18 00:08:22 2013
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sat May 18 00:08:22 2013
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Sat May 18 00:08:22 2013
SMON: enabling cache recovery
Sat May 18 00:08:22 2013
db_recovery_file_dest_size of 4096 MB is 8.15% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Sat May 18 00:08:23 2013
ARC2: Archival started
ARC1: STARTING ARCH PROCESSES COMPLETE
ARC1: Becoming the heartbeat ARCH
ARC2 started with pid=28, OS id=27166
Sat May 18 00:08:24 2013
Successfully onlined Undo Tablespace 5.
Sat May 18 00:08:24 2013
SMON: enabling tx recovery
Sat May 18 00:08:24 2013
Database Characterset is WE8ISO8859P1
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=30, OS id=27295
Sat May 18 00:08:30 2013
Completed: ALTER DATABASE OPEN
Sat May 18 00:09:37 2013
Shutting down instance (abort)
License high water mark = 6
Instance terminated by USER, pid = 29286



--asm alert log (  node2 ) 


Fri May 17 23:04:16 2013
Shutting down instance (abort)
License high water mark = 7
Instance terminated by USER, pid = 13696
Fri May 17 23:08:29 2013
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 192.168.2.0 configured from OCR for use as  a public interface
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2.0/asm_1/dbs/archAutotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  large_pool_size          = 12582912
  instance_type            = asm
  cluster_database         = TRUE
  instance_number          = 2
  remote_login_passwordfile= EXCLUSIVE
  background_dump_dest     = /u01/app/oracle/admin/+ASM/bdump
  user_dump_dest           = /u01/app/oracle/admin/+ASM/udump
  core_dump_dest           = /u01/app/oracle/admin/+ASM/cdump
  asm_diskgroups           = DATA, FRA
Cluster communication is configured to use the following interface(s) for this instance
  10.10.10.32
Fri May 17 23:08:31 2013
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=10304
LMON started with pid=5, OS id=10310
DIAG started with pid=3, OS id=10306
PSP0 started with pid=4, OS id=10308
LMS0 started with pid=7, OS id=10314
MMAN started with pid=8, OS id=10324
DBW0 started with pid=9, OS id=10326
LMD0 started with pid=6, OS id=10312
LGWR started with pid=10, OS id=10328
CKPT started with pid=11, OS id=10330
SMON started with pid=12, OS id=10338
RBAL started with pid=13, OS id=10342
Fri May 17 23:08:32 2013
lmon registered with NM - instance id 2 (internal mem no 1)
GMON started with pid=14, OS id=10348
Fri May 17 23:08:32 2013
Reconfiguration started (old inc 0, new inc 6)
ASM instance
List of nodes:
 0 1
 Global Resource Directory frozen
 Communication channels reestablished
* allocate domain 1, invalid = TRUE
 * domain 1 valid = 1 according to instance 0
* allocate domain 2, invalid = TRUE
 * domain 2 valid = 1 according to instance 0
Fri May 17 23:08:33 2013
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Fri May 17 23:08:33 2013
 LMS 0: 0 GCS shadows cancelled, 0 closed
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
Fri May 17 23:08:33 2013
 LMS 0: 0 GCS shadows traversed, 0 replayed
Fri May 17 23:08:33 2013
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=15, OS id=10369
Fri May 17 23:08:35 2013
SQL> ALTER DISKGROUP ALL MOUNT
Fri May 17 23:08:35 2013
NOTE: cache registered group DATA number=1 incarn=0xdb2871cb
NOTE: cache registered group FRA number=2 incarn=0xdb5871cc
Fri May 17 23:08:35 2013
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Fri May 17 23:08:35 2013
NOTE: Hbeat: instance not first (grp 1)
Fri May 17 23:08:35 2013
NOTE: Hbeat: instance not first (grp 2)
NOTE: cache opening disk 0 of grp 1: VOL1 label:VOL1
Fri May 17 23:08:35 2013
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0xDB2871CB (DATA)
Fri May 17 23:08:35 2013
kjbdomatt send to node 0
Fri May 17 23:08:36 2013
NOTE: attached to recovery domain 1
Fri May 17 23:08:37 2013
NOTE: opening chunk 2 at fcn 0.3707 ABA
NOTE: seq=6 blk=19
Fri May 17 23:08:37 2013
NOTE: cache mounting group 1/0xDB2871CB (DATA) succeeded
SUCCESS: diskgroup DATA was mounted
NOTE: cache opening disk 0 of grp 2: VOL2 label:VOL2
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0xDB5871CC (FRA)
Fri May 17 23:08:38 2013
kjbdomatt send to node 0
Fri May 17 23:08:38 2013
NOTE: recovering COD for group 1/0xdb2871cb (DATA)
SUCCESS: completed COD recovery for group 1/0xdb2871cb (DATA)
Fri May 17 23:08:39 2013
NOTE: attached to recovery domain 2
Fri May 17 23:08:40 2013
NOTE: opening chunk 2 at fcn 0.1733 ABA
NOTE: seq=6 blk=22
Fri May 17 23:08:40 2013
NOTE: cache mounting group 2/0xDB5871CC (FRA) succeeded
SUCCESS: diskgroup FRA was mounted
Fri May 17 23:08:41 2013
NOTE: recovering COD for group 2/0xdb5871cc (FRA)
SUCCESS: completed COD recovery for group 2/0xdb5871cc (FRA)
Fri May 17 23:08:51 2013
Starting background process ASMB
ASMB started with pid=17, OS id=10883
Fri May 17 23:11:49 2013
NOTE: ASMB process exiting due to lack of ASM file activity
Sat May 18 00:08:04 2013
Starting background process ASMB
ASMB started with pid=19, OS id=26613
Sat May 18 00:09:38 2013
Shutting down instance (abort)
License high water mark = 9
Instance terminated by USER, pid = 29333
[oracle@crs2 bdump]$