coreadm: dumpadm: Crash Dump Defined

A core file is a point-in-time copy (snapshot) of the RAM allocated to a process.

Two types of core files:

1)Per-process core file - owned by user mode 600

2)Global core file - not created by default ; owned by root mode 600

To change the Core File Configuration use the coreadm command to modify /etc/coreadm.conf file.

 # coreadm -p pattern pid
 # coreadm -i pattern ;survives reboot
 # coreadm -e global/process/global-setid/proc-setid/log ;enable options
 # coreadm -d ;disables corefile option
 # coreadm -u ;updates
 # coreadm -g ;sets the global core file name pattern

 %p ; PID
 %u ; uid
 %g ; gid
 %f ; executable filename
 %n ; system node name uname -n
 %m ; machine hardware name = uname -m
 %t ; time in seconds since 1970,1,1.
 %d ; executable file directory/name
 %z ; zonename
 %% ; % itself

Examples:

 # coreadm -p core.%f.%p $$ ; $$ pid of the current shell
 # coreadm -p $HOME/corefiles/%n.%f.%p $$
 # coreadm -g /var/core/core.%f.%p -e global ;
 # coreadm 228 507
 # coreadm -p /var/core/usr/bin ; listing corefiles for pid
 # coreadm -G all -g /var/core/%d/%f %p %n

The dumpadm, stores a dump of memory, the /etc/dumpadm.conf stores coredump configurations. Don't edit this file. use dumpadm command instead.

The default dump area is the swap space

/var/crash/host-name/vmcore.0 - memory content

/var/crash/host-name/unix.0 - symbol table

savecore command is a utility that saves crushdump into a file on reboot.

 # dumpadm
 # dumpadm -c curproc -d swap ; Modify the dump configuration so it dumps Kernel memory pages and its
 processes to swap memory
 # dumpadm -n ; turn off savecore
 # dumpadm -u ; update kernel from /etc/dumpadm.conf
 # dumpadm -y ; turn on savecore, default
 # dumpadm -c ; specify the dump content - kernel,all or curproc
 # dumpadm -d ; specify the dump device
 # dumpadm -m minK ; set a mininum space savecore should reserve,normally,in /var/crash/host1/
 filesystem.
 # dumpadm -s savecore_dir ; specify the directory where savecore saves
 # dumpadm -r root_dir ; specify the relative root dir, default /

dumpadm Examples Tests

 # dumpadm
 Dump content: kernel pages
 Dump device: /dev/dsk/c0t0d0s1 (swap)
 Savecore directory: /var/crash/host-name
 Savecore enabled: yes

 # dumpadm -d /dev/dsk/c1t0d0s5
 Dump content: kernel pages
 Dump device: /dev/dsk/c1t0d0s5 (dedicated)
 Savecore directory: /var/crash/host-name
 Savecore enabled: yes

 # sync

 # savecore -L
 dumping to /dev/dsk/c1t0d0s5, offset 65536, content: kernel
 100% done: 11679 pages dumped, compression ratio 3.14, dump succeeded 

 System dump time: Tue Dec 5 13:21:05 2006
 Constructing namelist /var/crash/host-name/unix.0
 Constructing corefile /var/crash/host-name/vmcore.0

 100% done: 11679 of 11679 pages saved

 # cd /var/crash/host-name
 -rw-r--r-- 1 root root 1201176 Dec 5 13:21 unix.0
 -rw-r--r-- 1 root root 97640448 Dec 5 13:21 vmcore.0

 # file vmcore.0
 vmcore.0: SunOS 5.10 Generic_118822-25 64-bit SPARC crash dump from host-name

coreadm Examples Tests

 # coreadm
 global core file pattern:
 global core file content: default
 init core file pattern: core
 init core file content: default
 global core dumps: disabled
 per-process core dumps: enabled
 global setid core dumps: disabled
 per-process setid core dumps: disabled
 global core dump logging: disabled

 # mkdir /var/core

 # coreadm -e global -g /var/core/core.%f.%p

 # coreadm -e log

 # coreadm
 global core file pattern: /var/core/core.%f.%p
 global core file content: default
 init core file pattern: core
 init core file content: default
 global core dumps: enabled
 per-process core dumps: enabled
 global setid core dumps: disabled
 per-process setid core dumps: disabled
 global core dump logging: enabled

 # mkdir /var/tmp/dir

 # cd /var/tmp/dir

 # pwd
 /var/tmp/dir

  # ps
 PID TTY TIME CMD
 1094 pts/6 0:00 ps
 1056 pts/6 0:00 ksh

 # kill 8 1056

 # ls /var/core
 core.ksh.729
 core.ksh.893
 /var/core/core.ksh.729: ELF 32-bit MSB core file SPARC Version 1, from 'ksh'
 /var/core/core.ksh.893: ELF 32-bit MSB core file SPARC Version 1, from 'ksh'

 # tail /var/adm/messages
 Dec 5 13:21:08 host-name genunix: [ID 851671 kern.notice] dump succeeded
 Dec 5 13:21:29 host-name savecore: [ID 748169 auth.error] saving system crash dumpin /var/crash
 /host-name/*.0
 Dec 5 13:26:29 host-name genunix: [ID 603404 kern.notice] NOTICE: core_log: ksh[893] core dumped:
 /var/core/core.ksh.893
 Dec 5 13:27:23 host-name genunix: [ID 603404 kern.notice] NOTICE: core_log: ksh[729] core dumped:
 /var/core/core.ksh.729

news: Oracle Solaris 10 Update 9 released

September 8, 2010 Oracle officially announced Oracle Solaris 10 9/10, Oracle Solaris Cluster 3.3 and Oracle Solaris Studio12.2. For now we will be concentrating on Oracle Solaris 10 9/10. 

So what Solaris 10 U9 includes, let’s take quick tour -

There are some drastic changes under this update -

• The most awaited, Oracle Solaris Containers now provide enhanced “P2V” (Physical to Virtual) capabilities to allow customers to seamlessly move from existing Oracle Solaris 10 physical systems to virtual containers quickly and easily. At our project we developed a custom method to perform p2v from Solaris8/9 to Solaris 10 Container and to be honest we was really looking forward for this feature.
• Host ID Emulation - Migration of a physical Solaris 10 machine into a Zone with support for the HostID will allow more network management platforms to be virtualized while still retaining their licensing features.
• Oracle 11g Release 2 Support
• Networking and database optimizations for Oracle Real Application Clusters (Oracle RAC).
• Increased reliability for virtualized Solaris instances when deployed using Oracle VM for SPARC, also known as Logical Domains.

ZFS Changes -

• ZFS device replacement enhancements - namely autoexpand
• some changes to the zpool list command
• Holding ZFS snapshots
• Triple parity RAID-Z (raidz3)
• The logbias property
• Log device removal - at last
• ZFS storage pool recovery
• New ZFS system process – In this release, each storage pool has an associated process, zpool-poolname
• Splitting a mirrored ZFS storage pool (zpool split)

For more information - http://dlc.sun.com/pdf/821-1840/821-1840.pdf

Download - http://www.oracle.com/technetwork/server-storage/solaris/downloads/index.html

XSCF: Upgrade of XCP firmware

This section explains how to update the firmware on Sun Sparc Enterpise M Series.

The steps will follow the preferred method from upgrading from 1060 firmware to 1093 firmware.The three major steps are :

- shut down to the ok prompt (init 0)
- XCP import in the system
- upgrade the XCP firmware - This will include an XSCF reset
- boot the system

Note – XCP: Abbreviation for XSCF Control Package. XCP is a package that has the control programs of hardware that configures a computing system. The XSCF firmware and the OpenBoot PROM firmware are included in the XCP file. The firmware update functions provided by XSCF are used to manage XCP.

Firmware update using the XSCF Shell, Use the following commands to update the firmware:
■ getflashimage command: Imports firmware to this system.
■ flashupdate command: Downloads the firmware to flash memory and applies the XSCF firmware.
■ poweron command or reset command: Applies the OpenBoot PROM firmware.
■ version command: Displays the firmware version.

1. Once you have the system shutdown to the ok prompt , enter into the XSCF prompt.

2. Before updating the firmware, be sure to check the XCP version in the current system. Be aware of which version your upgrading from as steps will differ if there is a large version gap.

 XSCF> version -c xcp -v
 XSCF#0 (Active )
 XCP0 (Current): 1060
 OpenBoot PROM : 01.30.0000
 XSCF          : 01.06.0001
 XCP1 (Reserve): 1060
 OpenBoot PROM : 01.30.0000
 XSCF          : 01.06.0001

3. Confirm the list of the firmware program files that are still on the system using the getflashimage command.

 XSCF> getflashimage -l
 Existing versions:
 Version                Size  Date
 FFXCP1060.tar.gz   49053148  Tue Feb 26 19:29:49 EST 2008

4. Use the following getflashimage command to specify the firmware program file and import XCP to the system.
Login a remote ftp server specifying the user name and host name that requires authentication password, then, import the new version firmware program (tar.gz). Ensure that your firmware program file is located in the home directory of the user your going to connect as. 

 XSCF> getflashimage -u user-name ftp://ip-address/FFXCP1093.tar.gz
 Existing versions:
 Version                Size  Date
 FFXCP1060.tar.gz   49053148  Tue Feb 26 19:29:49 EST 2008
 Warning: About to delete existing versions.
 Continue? [y|n]: y
 Removing FFXCP1060.tar.gz.
 Password:
   0MB received
   1MB received
   2MB received
 .......
  39MB received
  40MB received
 Download successful: 41859 Kbytes in 56 secs (784.888 Kbytes/sec)
 Checking file...
 MD5: f2dc08a4bd43061ea84c0172a6380c94

5. Confirm the list of the firmware program file you downloaded is now on the system using the getflashimage command.

 XSCF> getflashimage -l
 Existing versions:
 Version                Size  Date
 FFXCP1093.tar.gz   42863796  Thu Sep 23 14:09:40 EST 2010

6. Use the flashupdate command to confirm whether your able to update the new firmware version.

 XSCF> flashupdate -c check -m xcp -s 1093
 XCP update is possible with domains up

7. Use the flashupdate command to update the firmware. Once complete the the XSCF will reset and the current session will disconnect, connect again once the XSCF has been restored.

 XSCF> flashupdate -c update -m xcp -s 1093
 The XSCF will be reset. Continue? [y|n] :y
 Checking the XCP image file, please wait a minute
 XCP update is started (XCP version=1093:last version=1060)
 OpenBoot PROM update is started (OpenBoot PROM version=02160000)
 OpenBoot PROM update has been completed (OpenBoot PROM version=02160000)
 XSCF update is started (XSCFU=0,bank=1,XCP version=1093:last version=1060)
 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element
 ID=00:version=01090003:last version=01060000)
 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware
 Element ID=00:version=01090003:last version=01060000)
 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element
 ID=01:version=01090003:last version=01060001)
 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware
 Element ID=01:version=01090003:last version=01060001)
 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element
 ID=02:version=01080001:last version=01060000)
 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware
 Element ID=02:version=01080001:last version=01060000)
 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element
 ID=03:version=01090002:last version=01060000)
 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware
 Element ID=03:version=01090002:last version=01060000)
 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element
 ID=04:version=01090003:last version=01060001)
 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware
 Element ID=04:version=01090003:last version=01060001)
 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element
 ID=05:version=01090002:last version=01050000)
 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware
 Element ID=05:version=01090002:last version=01050000)
 XSCF download is started (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware Element
 ID=07:version=01090001:last version=01060000)
 XSCF download has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060, Firmware
 Element ID=07:version=01090001:last version=01060000)
 XSCF update has been completed (XSCFU=0,bank=1,XCP version=1093:last version=1060)
 XSCF is rebooting to update the reserve bank

8. Re-connect to the XSCF and log in again. To confirm that the XSCF firware update has finished, use the showlogs command with the monitor option. Ensure you see the "SCF:XCP update has been completed version=xxxx" message

 XSCF> showlogs monitor
 Sep 23 14:15:10 xscf1 monitor_msg: SCF:XCP update is started (XCP version=1093:last version=1060)
 Sep 23 14:15:49 xscf1 monitor_msg: SCF:XSCF download is started (XSCFU=0, bank=1, XCP
 version=1093:last version=1060, Firmware Element ID=00, version=01090003:last version=01060000)
 Sep 23 14:16:28 xscf1 monitor_msg: SCF:XSCF download has been completed (XSCFU=0, bank=1, XCP
 version=1093:last version=1060, Firmware Element ID=00, version=01090003:last version=01060000)
 Sep 23 14:16:41 xscf1 monitor_msg: SCF:XSCF download is started (XSCFU=0, bank=1, XCP
 version=1093:last version=1060, Firmware Element ID=01, version=01090003:last version=01060001)
 .......
 Sep 23 14:32:55 xscf1 monitor_msg: SCF:XCP update has been completed (XCP version=1093)

9. Confirm the version of the system firmware that is running is that of the firmware applied.

 XSCF> version -c xcp -v
 XSCF#0 (Active )
 XCP0 (Reserve): 1093
 OpenBoot PROM : 02.16.0000
 XSCF          : 01.09.0003
 XCP1 (Current): 1093
 OpenBoot PROM : 02.16.0000
 XSCF          : 01.09.0003
 OpenBoot PROM BACKUP
 #0: 01.30.0000
 #1: 02.16.0000

10. To complete the update restart the domain.  Once the domain is running it will commence its boot sequence.

 XSCF> reset -d 0 por
 DomainID to reset:00
 Continue? [y|n] :y
 00 :Reset
 XSCF> showdomainstatus -a
 DID         Domain Status
 00          Initialization Phase
 01          -
 02          -
 03          -
 XSCF> showdomainstatus -a
 DID         Domain Status
 00          Running
 01          -
 02          -
 03          -

timezones: Daylight Savings

Daylight Savings for this year starts on Sunday October 3. Are your systems ready to handle the time change automatically?

To check any systems you may be interested in, the easiest way is using the zdump command

 # zdump -v $TZ | grep 2010

 Australia/NSW  Thu Sep 23 05:08:57 2010 UTC = Thu Sep 23 15:08:57 2010 EST isdst=0
 Australia/NSW  Sat Apr  3 15:59:59 2010 UTC = Sun Apr  4 02:59:59 2010 EST isdst=1
 Australia/NSW  Sat Apr  3 16:00:00 2010 UTC = Sun Apr  4 02:00:00 2010 EST isdst=0
 Australia/NSW  Sat Oct  2 15:59:59 2010 UTC = Sun Oct  3 01:59:59 2010 EST isdst=0
 Australia/NSW  Sat Oct  2 16:00:00 2010 UTC = Sun Oct  3 03:00:00 2010 EST isdst=1

The first line is just the current date/time and can be ignored. the remaining entries show the entries from the time zone database, which determine when any time changes for this year will occur. The above shows that the system clock will go forward by 1 hour at 2:00 on Sunday Oct 3, which is correct.

veritas: Backing up the Veritas Cluster Server configuration

Veritas cluster server stores custom agents and it’s configuration data as a series of files in /etc, /etc/VRTSvcs/conf/config and /opt/VRTSvcs/bin/ directories. Since these files are the life blood of the cluster engine, it is important to backup these files to ensure cluster recovery should disaster hit. VCS comes with the hasnap utility to simplify cluster configuration backups, and when run with the “-backup,” “-n,” “-f ,” and “-m ” options, a point in time snapshot of the cluster configuration will be written to the file passed to the “-f” option:

 # hasnap -backup -f clusterbackup.zip -n -m “Backup from March 25th 2007″ 

 Starting Configuration Backup for Cluster foo

 Dumping the configuration...

 Registering snapshot "foo-2006.08.25-1156511358610"

 Contacting host lnode1...

 Error connecting to the remote host "lnode1"

 Starting backup of files on host lnode2
 "/etc/VRTSvcs/conf/config/types.cf" ----> 1.0
 "/etc/VRTSvcs/conf/config/main.cf" ----> 1.0
 "/etc/VRTSvcs/conf/config/vcsApacheTypes.cf" ----> 1.0
 "/etc/llthosts" ----> 1.0
 "/etc/gabtab" ----> 1.0
 "/etc/llttab" ----> 1.0
 "/opt/VRTSvcs/bin/vcsenv" ----> 1.0
 "/opt/VRTSvcs/bin/LVMVolumeGroup/monitor" ----> 1.0
 "/opt/VRTSvcs/bin/LVMVolumeGroup/offline" ----> 1.0
 "/opt/VRTSvcs/bin/LVMVolumeGroup/online" ----> 1.0
 "/opt/VRTSvcs/bin/LVMVolumeGroup/clean" ----> 1.0
 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0
 "/opt/VRTSvcs/bin/LVMVolumeGroup/LVMVolumeGroup.xml" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/fdsched" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/monitor" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/fdsetup.vxg" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/open" ----> 1.0
 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/RVGSnapshotAgent.pm" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/RVGSnapshot.xml" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/offline" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/online" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/attr_changed" ----> 1.0
 "/opt/VRTSvcs/bin/RVGSnapshot/clean" ----> 1.0
 "/opt/VRTSvcs/bin/RVGPrimary/monitor" ----> 1.0
 "/opt/VRTSvcs/bin/RVGPrimary/open" ----> 1.0
 "/opt/VRTSvcs/bin/RVGPrimary/RVGPrimary.xml" ----> 1.0
 "/opt/VRTSvcs/bin/RVGPrimary/offline" ----> 1.0
 "/opt/VRTSvcs/bin/RVGPrimary/online" ----> 1.0
 "/opt/VRTSvcs/bin/RVGPrimary/clean" ----> 1.0
 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0
 "/opt/VRTSvcs/bin/RVGPrimary/actions/fbsync" ----> 1.0
 "/opt/VRTSvcs/bin/triggers/violation" ----> 1.0
 "/opt/VRTSvcs/bin/CampusCluster/monitor" ----> 1.0
 "/opt/VRTSvcs/bin/CampusCluster/close" ----> 1.0
 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0
 "/opt/VRTSvcs/bin/CampusCluster/open" ----> 1.0
 "/opt/VRTSvcs/bin/CampusCluster/CampusCluster.xml" ----> 1.0
 "/opt/VRTSvcs/bin/RVG/monitor" ----> 1.0
 "/opt/VRTSvcs/bin/RVG/info" ----> 1.0
 "/opt/VRTSvcs/bin/ScriptAgent" ----> 1.0
 "/opt/VRTSvcs/bin/RVG/RVG.xml" ----> 1.0
 "/opt/VRTSvcs/bin/RVG/offline" ----> 1.0
 "/opt/VRTSvcs/bin/RVG/online" ----> 1.0
 "/opt/VRTSvcs/bin/RVG/clean" ----> 1.0
 "/opt/VRTSvcs/bin/internal_triggers/cpuusage" ----> 1.0

 Backup of files on host lnode2 complete

 Backup succeeded partially

To check the contents of the snapshot, the unzip utility can be run with the “-t” option:

 # unzip -t clusterbackup.zip |more
 Archive: clusterbackup.zip
 testing: /cat_vcs.zip OK
 testing: /categorylist.xml.zip OK
 testing: _repository__data/vcs/foo/lnode2/etc/VRTSvcs/conf/config/types.cf.zip OK
 testing: _repository__data/vcs/foo/lnode2/etc/VRTSvcs/conf/config/main.cf.zip OK
 testing: _repository__data/vcs/foo/lnode2/etc/VRTSvcs/conf/config/vcsApacheTypes.cf.z ip OK
 testing: _repository__data/vcs/foo/lnode2/etc/llthosts.zip OK
 testing: _repository__data/vcs/foo/lnode2/etc/gabtab.zip OK
 testing: _repository__data/vcs/foo/lnode2/etc/llttab.zip OK
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/vcsenv.zip OK
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/monitor.zip OK
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/offline.zip OK
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/online.zip OK
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/clean.zip OK
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/LVMVolumeGro upAgent.zip OK
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/LVMVolumeGroup/LVMVolumeGro up.xml.zip OK
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/RVGSnapshot/fdsched.zip O K
 testing: _repository__data/vcs/foo/lnode2/opt/VRTSvcs/bin/RVGSnapshot/monitor.zip O K
 ......

Since parts of the cluster configuration ran reside in memory and not on disk, it is a good idea to run “haconf -dump -makero” prior to running hasnap. This will ensure that the current configuration is being backed up, and will allow hasnap “-restore” to restore the correct configuration if disaster hits.

veritas: Veritas Volume Replicator (VVR) Commands

Here are some links to Basic Veritas Volume Replicator (VVR) commands for your data replication management environments.

Basic VVR Commands:

veritas: rlinks in 'paused due to network disconnection' state

Had a case open with Symantec with the following issue.

 # vradmin -g dgprdap2 repstatus rvg_prdap2
 Replicated Data Set: rvg_prdap2
 Primary:
   Host name:                  10.150.150.13
   RVG name:                   rvg_prdap2
   DG name:                    dgprdap2
   RVG state:                  enabled for I/O
   Data volumes:               1
   VSets:                      0
   SRL name:                   srl_prdap2
   SRL size:                   25.00 G
   Total secondaries:          1

 Secondary:
   Host name:                  10.150.150.16
   RVG name:                   rvg_prdap2
   DG name:                    dgprdap2
   Data status:                consistent, behind
   Replication status:         paused due to network disconnection
   Current mode:               asynchronous
   Logging to:                 SRL (213278 Kbytes behind, 0% full)
   Timestamp Information:      behind by 2668h 22m 7s

 'vradmin repstatus' shows rlinks in 'paused due to network disconnection' state

Various attempts where made to try continue/attach or resume the replication but failed.  

The following checks must be made to diagnose the problem.

Run the vrport command on both the primary and secondary nodes.

# /usr/sbin/vrport

Check communication on these ports, from primary to secondary and vice versa on each node: 

# ping -p port-number host-name

If the ping command succeeds, try restarting the VVR daemons (both on primary and secondary) in the correct sequence: 

 

Stop vradmin on secondary then on primary

# /usr/sbin/vxstart_vvr stop

Start vradmin on secondary then on primary

# /usr/sbin/vxstart_vvr start 

If for any unknown reason the above does not fix the issue you might need to try to restart the primary and secondary nodes. But if you encounter that a bounce does not fix issue it's a rare possibility that you might need to re-create the rvg's as it was in my case. Another unsolved VVR mystery.