I had trouble starting ASM proxy instance on one of the nodes in my 2 node flex cluster having nodes host01 and host02. As a result I could not access the volume I created on an ASM diskgroup. This post explains how I resolved it.
While connected to host01, I created a volume VOL1 on DATA diskgroup with corresponding volume device /dev/asm/vol1-106 .
[grid@host01 root]$ asmcmd volcreate -G DATA -s 300m VOL1
[grid@host01 root]$ asmcmd volinfo -G DATA VOL1
Diskgroup Name: DATA
Volume Name: VOL1
Volume Device: /dev/asm/vol1-106
State: ENABLED
Size (MB): 320
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS Mountpath:
I created ACFS file system on the newly created volume
[root@host01 ~]# mkfs -t acfs /dev/asm/vol1-106
I also created corresponding mount point /mnt/acfsmounts/acfs1 on both the nodes in the cluster.
root@host01 ~]# mkdir -p /mnt/acfsmounts/acfs1
root@host02 ~]# mkdir -p /mnt/acfsmounts/acfs1
When I tried to mount the volume device, I could mount the volume device on host01 but not on host02 .
[root@host01 ~]#mount -t acfs /dev/asm/vol1-106 /mnt/acfsmounts/acfs1
[root@host01 ~]# mount | grep vol1
/dev/asm/vol1-106 on /mnt/acfsmounts/acfs1 type acfs (rw)
[root@host02 ~]# mount -t acfs /dev/asm/vol1-106 /mnt/acfsmounts/acfs1
mount.acfs: CLSU-00100: Operating System function: open64 failed with error data: 2
mount.acfs: CLSU-00101: Operating System error message: No such file or directory
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/vol1-106) mount.acfs: ACFS-02017: Failed to open volume /dev/asm/vol1-106. Verify the volume exists.
The corresponding volume device was visible on host01 but not on host02
[root@host01 ~]# cd /dev/asm
[root@host01 asm]# ls
vol1-106
[root@host02 ~]# cd /dev/asm
[root@host02 asm]# ls
Since ADVM / ACFS utilize an ASM Proxy instance in a flex cluster to access metadata from a local / remote ASM instance , I checked whether ASM Proxy instance was running on both the nodes and realized that whereas ASM Proxy instance was running on host01, it was not running on host02
[root@host01 ~]# ps -elf | grep pmon | grep APX0 S grid 27782 1 0 78 0 – 350502 – 10:09 ? 00:00:00 apx_pmon_+APX1
[root@host02 asm]# ps -elf | grep pmon | grep APX
[root@host01 ~]# srvctl status asm -proxy
ADVM proxy is running on node host01
[root@host01 ~]# crsctl stat res ora.proxy_advm -t——————————————————————————– Name Target State Server State details——————————————————————————– Local Resources ——————————————————————————–
ora.proxy_advm
ONLINE ONLINE host01 STABLE
ONLINE OFFLINE host02 STABLE
I tried to start ASM proxy instance manually on host02
[grid@host02 ~]$ . oraenv
ORACLE_SID = [grid] ? +APX2
The Oracle base has been set to /u01/app/grid
[grid@host02 ~]$ sqlplus / as sysasm
SQL*Plus: Release 12.1.0.1.0 Production on Sat May 2 10:31:45 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-00099: warning: no parameter file specified for ASMPROXY instance
ORA-00443: background process "VUBG" did not start
SQL> ho oerr ORA 00443
00443, 00000, "background process \"%s\" did not start"
// *Cause: The specified process did not start.
// *Action: Ensure that the executable image is in the correct place with
// the correct protections, and that there is enough memory.
I checked the memory allocated to VM for host02 – It was 1.5 GB as against 2.5 GB assigned to VM for host01. I increased the memory of host02 to 2.5 GB and ASM proxy instance started automatically.
[root@host01 ~]# crsctl stat res ora.proxy_advm -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.proxy_advm
ONLINE ONLINE host01 STABLE
ONLINE ONLINE host02 STABLE