java: Alert on java crashes

The last couple of days I have noticed a number of irregular core dumps created in my system core file dump location /var/core

 -rw-------   1 root    root  2529790203 Nov 10 11:55 core_host1_java_1094_300_1289350401_28578
 -rw-------   1 root    root  2564932547 Nov 15 13:06 core_host1_java_1094_300_1289786684_1664
 -rw-------   1 root    root  2498732827 Nov 17 17:29 core_host1_java_9092_300_1289975232_5664
 -rw-------   1 root    root  2525420387 Nov 19 12:08 core_host3_java_1094_300_1290128885_16234

Depending on how you've setup your core file dump pattern, you can determine which process/application user account its comming from by just reading the file core name. eg

 # coreadm|grep pattern
    global core file pattern: /var/core/core_%n_%f_%u_%g_%t_%p
    init core file pattern: /var/core/core_%n_%f_%u_%g_%t_%p

 %n ; system node name uname -n
 %f ; executable filename
 %u ; uid
 %g ; gid
 %t ; time in seconds since 1970,1,1.
 %p ; PID

 
My core dump process is coming from a java process. Bugs can occur in a Java runtime environment and most administrators would want to get notified.
If you need to take a corrective action and diagnose further, you will need to be alerted at the time of incident.
The Java runtime has a number of useful options that can be used for this purpose. The first option is “-XX:OnOutOfMemoryError”, which allows a command to be run when the runtime environment incurs an out of memory condition. When this option is combined with the logger command line utility:

 java -XX:OnOutOfMemoryError=”logger Java process %p encountered an OOM condition” …

Syslog entries will be generated each time an Out Of memory (OOM) event occurs.

Another useful option is “-XX:OnError”, which allows a command to be run when the runtime environment incurs a fatal error (i.e., a hard crash). When this option is combined with the logger utility:

 java -XX:OnError=”logger -p Java process %p encountered a fatal condition” …

Syslog entries will be generated when a fatal event occur.

The options above allow you to run one or more commands when these errors are encountered, so you could chain together a utility (logger or mail) to generate alerts, and maybe a restarter script to start a new Java process.

zones: Physical to Virtual (P2V) Migration

Since the release of Solaris 10 Update 9 I was interested in trying out its new capabilities. One of which was to move from an existing Oracle Solaris 10 Physical system to a virtual container quickly and easily on a separate system. And also the Host ID migration feature which it also allows.

In this post I will provide a demonstration of P2V , migrating a physical Solaris system into a zone, using Host ID migration, keeping in mind in this example both systems are built from a Solaris 10 Update 9 release and is a ZFS on root build.





1. Collect the system information you wish to be migrated into a zone.
Obtain the hostname:

 # hostname

Obtain the hostid:

 # hostid

 
Obtain the root passwd
View what software being run on the system if necessary
Check the network configuration on the system:

 # ifconfig -a   

View the storage utilized for example the contents of /etc/vfstab
View the amount of local disk storage in use, and determine the target host has enough space for the install
Examine the /etc/system of any specific or unique changes you may wish to keep or change once the  environment has been virtualized.

2. Use the flarcreate command to create a flash archive image of the system you wish to migrate.
Login as root to the source system and change to the root directory.

 # cd /

   
Run the flarcreate using the following options:
Note: Here I have used the option to compress using the -c , used the archived method with cpio, you can use pax if you wish, then supplied the content name identifier of the archive using the -n option (best practice s10u9-system-name), also I have used the -y option because this system has a separate /var dataset and is ZFS on root based, then supplied the path destination of the flar archive creation.
When the flar creation is running be sure to monitor and review any errors thoroughly.

 # flarcreate -c -L cpio -n name -y /var /path/to/flar

3. Transfer the flar archive created to the destination target host.

4. Create a new zone configuration on the target host:
Note: I don't want to inherit any packages from the global or any loop-back file systems, so I have created a whole root zone configuration and provided a new  ip-address as I don't want it to conflict with the existing migrated host as its still alive.

Also you will see I have added a hostid entry in the zone configuration, when applications are migrated from a physical Solaris system into a zone on a new system, the hostid changes to be the hostid of the new machine. In some cases, applications depend on the original hostid, and it is not possible to update the application configuration. In these cases, the zone can be configured to use the hostid of the original system. This is done by setting a zonecfg property to specify the hostid as shown below. The value used should be the output of the hostid command as run on the original system previously.

 # zonecfg -z hostname
 hostname: No such zone configured
 Use 'create' to begin configuring a new zone.
 zonecfg:hostname> create -b
 zonecfg:hostname> set autoboot=true
 zonecfg:hostname> set zonepath=/zones/hostname
 zonecfg:hostname> set bootargs="-m verbose"
 zonecfg:hostname> set hostid=84###375
 zonecfg:hostname> add net
 zonecfg:hostname:net> set physical=bge0
 zonecfg:hostname:net> set address=ip-address
 zonecfg:hostname:net> end
 zonecfg:hostname> verify
 zonecfg:hostname> commit
 zonecfg:hostname> exit

5. Installing the zone on the target system using the flar archive created. Become root user and install the configured zone using the install -a option and the path to the flar archive created. Noticed I have used the -p option becuase I want to preserve the system identity, the zone will have the same identity as the system used to create the image. You can use the -u to sys-unconfig the zone.
Best practice make sure you tail the zone installation log file and ensure no errors are found.

 # zoneadm -z hostname install -p -a /path/to/flar
 A ZFS file system has been created for this zone.
       Log File: /var/tmp/hostname.install_log.IQaGnI
     Installing: This may take several minutes...
 Postprocessing: This may take a while...
    Postprocess: Updating the zone software to match the global zone...
    Postprocess: Zone software update complete
    Postprocess: Updating the image to run within a zone

         Result: Installation completed successfully.
       Log File: /zones/hostname/root/var/log/hostname.install17462.log

6. Boot the zone into single user mode and login via the console and of course use the root password from your migrated host. Make any necessary checks.

 # zoneadm -z hostname boot -s
 # zlogin -C hostname

From here you can see the P2V is complete, hostname and data has been kept intact including the host-id.
Now you can decide whether you need to make any further necessary changes such as hostname, network configuration etc etc. And of course be prepared to boot the migrated zone for a live production environment.

news: Oracle Solaris 11 Express Download Available

Solaris 11 Express 2010.11 is now available for download .
You will also find an overview and documentation available at the following link:
Click Here:

Solaris 11 Express now allows administrators to test and deploy within their enterprise environments and greatly simplify their day to day operations. It contains many different technology innovations that are not available in Oracle Solaris 10 such as new package management tools and utilities, built-in network virtualization, and support for the latest hardware platforms. To see the list of features that
are new to Oracle Solaris 11 Express please Clicke Here:

Oracle Solaris 11 Express is the latest release of the OracleSolaris operating system. This release is the path forward for developers, end-users and partners using previous generations of OpenSolaris releases.
This release will provide administrators to access the latest technology and innovation that will form
a future Oracle Solaris 11 which will be released sometime in 2011.

Here is a youtube video tutorial if you wish to test it out in your virtualbox setup. Enjoy
http://www.youtube.com/watch?v=r5hlrqlQAIc

inetd: Disable inetd Connection Logging for individual Services

Noticed a large number of unwanted constant connection messages in my system messages file.




 

 Aug 31 18:36:39 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19080] from ip-address 45632
 Aug 31 18:36:39 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19081] from ip-address 45633
 Aug 31 18:40:35 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19288] from ip-address 48640
 Aug 31 18:40:39 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19290] from ip-address 48641
 Aug 31 18:41:05 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19333] from ip-address 48653
 Aug 31 18:41:05 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19333] from ip-address 48653
 Aug 31 18:41:05 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19334] from ip-address 48654
 Aug 31 18:45:51 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19543] from ip-address 48714
 Aug 31 18:45:52 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19544] from ip-address 48715
 Aug 31 18:50:09 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19781] from ip-address 48786
 Aug 31 18:50:09 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[19782] from ip-address 48787
 Aug 31 18:57:59 ausydwebt01 inetd[455]: [ID 317013 daemon.notice] vnetd[24199] from ip-address 48871

The above is coming from my Veritas Netbackup network connection daemon which is constantly filling up my messages file during its nightly backup procedure.

If inetd is running, the "tracing" feature can be used to log information about the source of any network connections seen by the daemon. Rather than disabling inetd tracing for all services, the administrator has the option of disabling tracing for individual services with inetadm -m svcname tcp_trace=FALSE , where is the name of the specific service that should use tracing.

1. The following command will display the properties for the vnetd service.

 # inetadm -l svc:/network/vnetd/tcp:default
 SCOPE    NAME=VALUE
          name="vnetd"
          endpoint_type="stream"
          proto="tcp"
          isrpc=FALSE
          wait=FALSE
          exec="/usr/openv/bin/vnetd"
          user="root"
 default  bind_addr=""
 default  bind_fail_max=-1
 default  bind_fail_interval=-1
 default  max_con_rate=-1
 default  max_copies=-1
 default  con_rate_offline=-1
 default  failrate_cnt=40
 default  failrate_interval=60
 default  inherit_env=TRUE
 default  tcp_trace=TRUE
 default  tcp_wrappers=TRUE
 default  connection_backlog=10

2. The following command will disable tracing for the vnetd service

 # inetadm -m svc:/network/vnetd/tcp:default tcp_trace=FALSE

3. Confirm the changes using the display option again.

 
 # inetadm -l svc:/network/vnetd/tcp:default
 SCOPE    NAME=VALUE
          name="vnetd"
          endpoint_type="stream"
          proto="tcp"
          isrpc=FALSE
          wait=FALSE
          exec="/usr/openv/bin/vnetd"
          user="root"
 default  bind_addr=""
 default  bind_fail_max=-1
 default  bind_fail_interval=-1
 default  max_con_rate=-1
 default  max_copies=-1
 default  con_rate_offline=-1
 default  failrate_cnt=40
 default  failrate_interval=60
 default  inherit_env=TRUE
          tcp_trace=FALSE
 default  tcp_wrappers=TRUE
 default  connection_backlog=10

news: Solaris 11 Express Summit

The slides are now available for the presentations at the Oracle Solaris 11 Express Summit at the LISA Conference which was hosted on Tuesday Nov 9th.
The event showcased Oracle Solaris 11 Express, targeting System Administrators and Architects.






 
Here are the slides available for the following presentations:

• Slide 1 Introduction to Oracle Solaris 11 Express, Markus Flierl
• Slide 2 Image Packaging System, Bart Smaalders
• Slide 3 Deploying Oracle Solaris 11 in the Enterprise, Dave Miner
• Slide 4 Advances in Solaris Networking with Crossbow and Beyond, Nicolas Droux
• Slide 5 Oracle Solaris Containers in Oracle Solaris 11 Express, Dan Price
• Slide 6 ZFS Features in Oracle Solaris Express, Cindy Swearingen
• Slide 7 New Security Features in Oracle Solaris 11 Express, Glenn Faden
• Slide 8 Deploying Applications Using SMF and Other Solaris 11 Features, Liane Praza

The stream of videos have been recorded and can be viewed at the following link: Click here

 

JASS: Auditing & Controlling Output Logs

You can configure the Solaris Security Toolkit audit option to report or omit banners and messages. 

You might want to eliminate pass messages (JASS_LOG_SUCCESS variable) from the output so you can report and focus only on fail messages (JASS_LOG_FAILURE variable).

If the logging variable is set to 0, then no output is generated for messages of that type. Conversely, if the logging variable is set to 1, then messages are displayed. The default action for each of these variables is to display the output.

All Banner Output, This parameter controls the display of banner messages. These messages are
usually surrounded by separators comprised of either equal sign (“=”) or dash (“-”)
characters.

 JASS_LOG_BANNER

[ERR], This parameter controls the display of error messages. If set to 0, no error messages will be generated.

 JASS_LOG_ERROR

[FAIL] This parameter controls the display of failure messages. If set to 0, no failure messages will be generated.

 JASS_LOG_FAILURE

[NOTE] This parameter controls the display of notice messages. If set to 0, no notice messages will be generated.

 JASS_LOG_NOTICE

[PASS] This parameter controls the display of success or passing status messages. If set to 0, no success messages will be generated.

 JASS_LOG_SUCCESS

[WARN] This parameter controls the display of warning messages. If set to 0, no warning messages will be generated.

 JASS_LOG_WARNING

Using these options is very useful when you only need to view specific messages. By setting these options, you can minimize output, yet still focus on areas you deem critical. For example, by setting all logging variables to 0 except for JASS_LOG_FAILURE (leave it at the default of 1), the audit reports only on failures
generated by the logFailure function.

 # JASS_LOG_FAILURE=1
 # JASS_LOG_FAILURE=1
 # JASS_LOG_NOTICE=0
 # JASS_LOG_SUCCESS=0
 # JASS_LOG_WARNING=0
 # export JASS_LOG_WARNING JASS_LOG_SUCCESS JASS_LOG_NOTICE JASS_LOG_FAILURE

 # ./jass-execute -a secure.driver -V 2
 update-at-deny [FAIL] User test is not listed in
 /etc/cron.d/at.deny.
 update-at-deny [FAIL] Audit Check Total : 1 Error(s)
 update-inetd-conf [FAIL] Service ftp is enabled in
 /etc/inet/inetd.conf.
 update-inetd-conf [FAIL] Service telnet is enabled in
 /etc/inet/inetd.conf.
 update-inetd-conf [FAIL] Service rstatd is enabled in
 /etc/inet/inetd.conf.
 update-inetd-conf [FAIL] Audit Check Total : 3 Error(s)

Here I have provided a Jass Auditing script that can be run on a weekly, monthly , yearly however you may choose. The audit will alert on any system changes via Email set by the MAIL_LIST variable. The script requires Repository directory under /opt/SUNWjass.

jass-audit.sh

 #!/usr/bin/ksh

 HOST=`hostname`
 TIMESTAMP=`date +%H%M.%d%m`
 SPOOL="/opt/SUNWjass"
 L_LOG="$SPOOL/Repository/Jass_Audit.$TIMESTAMP"
 L_OUT="$SPOOL/Repository/Jass_Audit.$TIMESTAMP.OUT"
 MAIL_LIST=""
 JASS_LOG_FAILURE=1
 JASS_LOG_NOTICE=0
 JASS_LOG_SUCCESS=0
 JASS_LOG_WARNING=0
 export JASS_LOG_WARNING JASS_LOG_SUCCESS JASS_LOG_NOTICE JASS_LOG_FAILURE

 $SPOOL/bin/jass-execute -a server-secure.driver -V 2 -o $L_LOG

 if [ -f $L_LOG ]; then
    ERR=`grep FAIL $L_LOG|wc -l`
     if [ $ERR -ne 0 ]; then
      echo "Solaris Security Log: AUDIT (${HOST}) $TIMESTAMP" > $L_OUT
      echo "" >> $L_OUT
      echo "File : "$L_LOG" " >> $L_OUT
      echo "========================================================"  >> $L_OUT
      echo "Failures : " >> $L_OUT
      grep FAIL $L_LOG | egrep -v Error >> $L_OUT
      echo "========================================================"  >> $L_OUT
      tail -12 $L_LOG >> $L_OUT
      mailx -s "Solaris Security Toolkit Log: AUDIT (${HOST})" $MAIL_LIST < $L_OUT
     else
      exit 0
     fi
 fi

Output Example:

 Solaris Security Log: AUDIT (host-name) 1301.0211

 File : /opt/SUNWjass/Repository/Jass_Audit.1301.0211

 ========================================================

 Failures :

 update-at-deny                 [FAIL] User test is not listed in /etc/cron.d/at.deny.

 ========================================================

 server-secure.driver           [SUMMARY] Results Summary for AUDIT run of server-secure.driver

 server-secure.driver           [SUMMARY] The run completed with a total of 84 scripts run.

 server-secure.driver           [SUMMARY] There was a Failure  in   1 Script

 server-secure.driver           [SUMMARY] There were  Errors   in   0 Scripts

 server-secure.driver           [SUMMARY] There was a Warning  in   1 Script

 server-secure.driver           [SUMMARY] There were  Notes    in  19 Scripts

 server-secure.driver           [SUMMARY] Failure Scripts listed in:

 server-secure.driver                   /var/opt/SUNWjass/run/20101102130155/jass-script-failures.txt

 server-secure.driver           [SUMMARY] Warning Scripts listed in:

 server-secure.driver                   /var/opt/SUNWjass/run/20101102130155/jass-script-warnings.txt

 server-secure.driver           [SUMMARY] Notes Scripts listed in:

 server-secure.driver                   /var/opt/SUNWjass/run/20101102130155/jass-script-notes.txt

Veritas: Displaying the Status of VVR Links

The status of VVR links can be displayed by running the folloing commands, as root, on the appropriate host.

• Log in to the primary host of the relevant VVR link and su to root
• To view the status of the link, run the command

 # /usr/sbin/vxrlink -g disk-group-name status rlink-name

• Or the following command to update every 5 seconds

 # /usr/sbin/vrlink -g disk-group-name -i 5 status rlink-name

• Another option to show link status is to use the following vradmin command. This command displays more information than the previous one, including SRL size, current mode, and the amount of time (if any) that replication is behind.

 # /usr/sbin/vradmin -g disk-group-name repstatus rvg-name

• To view the amount of traffic being sent over the network for a VVR Link, run the following command, notice how I use the option "stats" instead of "status" here.

 # /usr/sbin/vrlink -g disk-group-name -i 5 stats rlink-name

 

JASS: Enable Veritas Netbackup Services in JASS

Once you have installed JASS you will need to edit specific files so to customize based on individual site needs. 

In this case if you have installed Veritas Netbackup and your host is hardened with JASS you will experience issues with Netbackup services and its TCP wrappers.

You will need to tell JASS to enable and open all Netbackup services related deamons and ports.

From the SUNWjass folder you will need to edit the following files:

 ./Files/etc/hosts.allow-server:
 ./Drivers/server-secure.driver:
 ./Drivers/finish.init:

Grant access to the following Netbackup services, add the following lines to JASS ./Files/etc/hosts.allow-server file:

 bpcd:   ALL
 vnetd:  ALL
 vopied: ALL
 bpjava-msvc:    ALL

Update the Finish script and enable the following Netbackup Services. In the ./Drivers/finish.init  file locate the JASS_SVCS_ENABLE section and add the following services string in the if statement.

Before:

 if [ -z "${JASS_SVCS_ENABLE}" ]; then
    JASS_SVCS_ENABLE=""
 fi
 export JASS_SVCS_ENABLE/logs/error_log

After:

 if [ -z "${JASS_SVCS_ENABLE}" ]; then
    JASS_SVCS_ENABLE="
       svc:/network/vnetd/tcp:default
       svc:/network/bpjava-msvc/tcp:default
       svc:/network/bpcd/tcp:default
       svc:/network/vopied/tcp:default"
 fi
 export JASS_SVCS_ENABLE

Update the secure-driver script and also enable the following Netbackup services. In the ./Drivers/server-secure.driver locate the JASS_SVCS_ENABLE variable and add the Netbackup deamons and services

Before:

 JASS_SVCS_ENABLE="${JASS_SVCS_ENABLE} dtspc rstatd 100155
 svc:/network/rpc/rstat:default
 svc:/network/rpc/smserver:default "

After:

 JASS_SVCS_ENABLE="${JASS_SVCS_ENABLE} dtspc rstatd 100155 bpcd bpjava-msvc vnetd vopied
 svc:/network/rpc/rstat:default
 svc:/network/rpc/smserver:default
 svc:/network/vnetd/tcp:default
 svc:/network/bpjava-msvc/tcp:default
 svc:/network/bpcd/tcp:default
 svc:/network/vopied/tcp:default "

Run JASS hardening to complete the configuration changes.

 # /opt/SUNWjass/bin/jass-execute -d server-secure.driver

veritas: Disk mirroring using Veritas Volume Manager

This paper will present a short introduction to mirroring two disks using Veritas Volume Manager (VxVM)

In this case you will see that two separate set of Storage Array disks can be mirrored across a single disk group without interruptions.

The following vxprint shows the contents of disk group dgprdnfs. It has a number of small sets of disks and from a single controller/storage c3's . The disks a set across three sets of concatenated volumes volprdnfs1 volprdnfs2 & volprdnfs3. 

 # vxprint -qhtg dgprdnfs
 dg dgprdnfs     default      default  16000    1106535203.86.eap42

 dm c3t0d52      c3t0d52s2    auto     2048     53028096 -
 dm c3t0d53      c3t0d53s2    auto     2048     53028096 -
 dm c3t0d54      c3t0d54s2    auto     2048     53028096 -
 dm c3t0d55      c3t0d55s2    auto     2048     53028096 -
 dm c3t0d122     c3t0d122s2   auto     2048     53026176 -
 dm c3t0d123     c3t0d123s2   auto     2048     53026176 -
 dm c3t0d124     c3t0d124s2   auto     2048     53026176 -
 dm c3t0d125     c3t0d125s2   auto     2048     53026176 -
 dm c3t0d126     c3t0d126s2   auto     2048     53026176 -
 dm c3t0d127     c3t0d127s2   auto     2048     53026176 -
 dm c3t0d128     c3t0d128s2   auto     2048     53026176 -
 dm c3t0d129     c3t0d129s2   auto     2048     53026176 -
 dm c3t0d130     c3t0d130s2   auto     2048     53026176 -
 dm c3t0d131     c3t0d131s2   auto     2048     53026176 -
 dm c3t0d132     c3t0d132s2   auto     2048     53026176 -
 dm c3t0d133     c3t0d133s2   auto     2048     53026176 -
 dm c3t0d203     c3t0d203s2   auto     2048     53026176 -
 dm dgprdnfs01   c3t0d12s2    auto     2048     53028096 -
 dm dgprdnfs02   c3t0d13s2    auto     2048     53028096 -
 dm dgprdnfs03   c3t0d14s2    auto     2048     53028096 -
 dm dgprdnfs04   c3t0d18s2    auto     2048     53028096 NOHOTUSE

 v  volprdnfs1   -            ENABLED  ACTIVE   943718400 SELECT   -        fsgen
 pl volprdnfs1-01 volprdnfs1  ENABLED  ACTIVE   943718400 CONCAT   -        RW
 sd dgprdnfs01-01 volprdnfs1-01 dgprdnfs01 0    53028096 0         c3t0d12  ENA
 sd dgprdnfs02-01 volprdnfs1-01 dgprdnfs02 0    9886464  53028096  c3t0d13  ENA
 sd dgprdnfs02-03 volprdnfs1-01 dgprdnfs02 19772928 33255168 62914560 c3t0d13 ENA
 sd dgprdnfs04-01 volprdnfs1-01 dgprdnfs04 0    39544576 96169728  c3t0d18  ENA
 sd c3t0d52-01   volprdnfs1-01 c3t0d52 0        53028096 135714304 c3t0d52  ENA
 sd c3t0d53-01   volprdnfs1-01 c3t0d53 0        53028096 188742400 c3t0d53  ENA
 sd c3t0d54-01   volprdnfs1-01 c3t0d54 0        53028096 241770496 c3t0d54  ENA
 sd c3t0d55-01   volprdnfs1-01 c3t0d55 0        50630912 294798592 c3t0d55  ENA
 sd c3t0d126-02  volprdnfs1-01 c3t0d126 53024   48247168 345429504 c3t0d126 ENA
 sd c3t0d127-01  volprdnfs1-01 c3t0d127 0       53026176 393676672 c3t0d127 ENA
 sd c3t0d128-01  volprdnfs1-01 c3t0d128 0       53026176 446702848 c3t0d128 ENA
 sd c3t0d129-01  volprdnfs1-01 c3t0d129 0       53026176 499729024 c3t0d129 ENA
 sd c3t0d130-01  volprdnfs1-01 c3t0d130 0       53026176 552755200 c3t0d130 ENA
 sd c3t0d131-01  volprdnfs1-01 c3t0d131 0       53026176 605781376 c3t0d131 ENA
 sd c3t0d132-01  volprdnfs1-01 c3t0d132 0       53026176 658807552 c3t0d132 ENA
 sd c3t0d133-01  volprdnfs1-01 c3t0d133 0       53026176 711833728 c3t0d133 ENA
 sd dgprdnfs04-02 volprdnfs1-01 dgprdnfs04 39544576 13483520 764859904 c3t0d18 ENA
 sd c3t0d55-02   volprdnfs1-01 c3t0d55 50630912 1898016  778343424 c3t0d55  ENA
 sd c3t0d123-03  volprdnfs1-01 c3t0d123 132064  52894112 780241440 c3t0d123 ENA
 sd c3t0d124-03  volprdnfs1-01 c3t0d124 130320  52895856 833135552 c3t0d124 ENA
 sd c3t0d125-02  volprdnfs1-01 c3t0d125 65168   52961008 886031408 c3t0d125 ENA
 sd c3t0d126-03  volprdnfs1-01 c3t0d126 48300192 4725984 938992416 c3t0d126 ENA

 v  volprdnfs2   -            ENABLED  ACTIVE   115343360 SELECT   -        fsgen
 pl volprdnfs2-01 volprdnfs2  ENABLED  ACTIVE   115343360 CONCAT   -        RW
 sd dgprdnfs02-02 volprdnfs2-01 dgprdnfs02 9886464 9886464 0       c3t0d13  ENA
 sd dgprdnfs03-01 volprdnfs2-01 dgprdnfs03 0    53028096 9886464   c3t0d14  ENA
 sd c3t0d203-01  volprdnfs2-01 c3t0d203 0       52428800 62914560  c3t0d203 ENA

 v  volprdnfs3   -            ENABLED  ACTIVE   53026176 SELECT    -        fsgen
 pl volprdnfs3-01 volprdnfs3  ENABLED  ACTIVE   53026176 CONCAT    -        RW
 sd c3t0d122-01  volprdnfs3-01 c3t0d122 0       53026176 0         c3t0d122 ENA

To start the mirroring process I have been given a set of of new disks. These sets of disks need to be equal or large enough to complete the mirroring process. Notice how these disks are on a separate controller/storage c4's.

 # vxdisk -o alldgs list|grep c4
 c4t2d1s2     auto:none       -            -            online invalid
 c4t2d2s2     auto:none       -            -            online invalid
 c4t2d3s2     auto:none       -            -            online invalid

Add new disks for mirroring to the same disk group as source.

 # vxdg -g dgprdnfs adddisk c4t2d1=c4t2d1s2
 # vxdg -g dgprdnfs adddisk c4t2d2=c4t2d2s2
 # vxdg -g dgprdnfs adddisk c4t2d3=c4t2d3s2

 # vxdisk -o alldgs list|grep c4
 c4t2d1s2     auto:cdsdisk    c4t2d1       dgprdnfs     online
 c4t2d2s2     auto:cdsdisk    c4t2d2       dgprdnfs     online
 c4t2d3s2     auto:cdsdisk    c4t2d3       dgprdnfs     online

Execute and Create mirror specifying the new disks to mirror each volume.

 # vxassist -g dgprdnfs -b mirror volprdnfs1 alloc=c4t2d1
 # vxassist -g dgprdnfs -b mirror volprdnfs2 alloc=c4t2d2
 # vxassist -g dgprdnfs -b mirror volprdnfs3 alloc=c4t2d3

After the above steps the following vxprint shows the new set of c4 disks attached to dgprdnfs disk group. Notice the sectored size of each disk, which must equal or be larger than the total volume size will are mirroring.

With the above vxassist command along with the mirror option we have executed the mirroring process, notice it has created a new plex for each volume volprdnfs1-02 volprdnfs2-02 volprdnfs3-02, each new plex is in TEMPRMSD state , which means an attached new temporary plex state, its copying the volume contents to the existing new plex, it wont be considered enabled once the process has completed.

 # vxprint -qhtg dgprdnfs
 dg dgprdnfs     default      default  16000    1106535203.86.eap42

 dm c3t0d52      c3t0d52s2    auto     2048     53028096 -
 dm c3t0d53      c3t0d53s2    auto     2048     53028096 -
 dm c3t0d54      c3t0d54s2    auto     2048     53028096 -
 dm c3t0d55      c3t0d55s2    auto     2048     53028096 -
 dm c3t0d122     c3t0d122s2   auto     2048     53026176 -
 dm c3t0d123     c3t0d123s2   auto     2048     53026176 -
 dm c3t0d124     c3t0d124s2   auto     2048     53026176 -
 dm c3t0d125     c3t0d125s2   auto     2048     53026176 -
 dm c3t0d126     c3t0d126s2   auto     2048     53026176 -
 dm c3t0d127     c3t0d127s2   auto     2048     53026176 -
 dm c3t0d128     c3t0d128s2   auto     2048     53026176 -
 dm c3t0d129     c3t0d129s2   auto     2048     53026176 -
 dm c3t0d130     c3t0d130s2   auto     2048     53026176 -
 dm c3t0d131     c3t0d131s2   auto     2048     53026176 -
 dm c3t0d132     c3t0d132s2   auto     2048     53026176 -
 dm c3t0d133     c3t0d133s2   auto     2048     53026176 -
 dm c3t0d203     c3t0d203s2   auto     2048     53026176 -
 dm c4t2d1       c4t2d1s2     auto     2048     964654848 -
 dm c4t2d2       c4t2d2s2     auto     2048     125821696 -
 dm c4t2d3       c4t2d3s2     auto     2048     62909696 -
 dm dgprdnfs01   c3t0d12s2    auto     2048     53028096 -
 dm dgprdnfs02   c3t0d13s2    auto     2048     53028096 -
 dm dgprdnfs03   c3t0d14s2    auto     2048     53028096 -
 dm dgprdnfs04   c3t0d18s2    auto     2048     53028096 NOHOTUSE

 v  volprdnfs1   -            ENABLED  ACTIVE   943718400 SELECT   -        fsgen
 pl volprdnfs1-01 volprdnfs1  ENABLED  ACTIVE   943718400 CONCAT   -        RW
 sd dgprdnfs01-01 volprdnfs1-01 dgprdnfs01 0    53028096 0         c3t0d12  ENA
 sd dgprdnfs02-01 volprdnfs1-01 dgprdnfs02 0    9886464  53028096  c3t0d13  ENA
 sd dgprdnfs02-03 volprdnfs1-01 dgprdnfs02 19772928 33255168 62914560 c3t0d13 ENA
 sd dgprdnfs04-01 volprdnfs1-01 dgprdnfs04 0    39544576 96169728  c3t0d18  ENA
 sd c3t0d52-01   volprdnfs1-01 c3t0d52 0        53028096 135714304 c3t0d52  ENA
 sd c3t0d53-01   volprdnfs1-01 c3t0d53 0        53028096 188742400 c3t0d53  ENA
 sd c3t0d54-01   volprdnfs1-01 c3t0d54 0        53028096 241770496 c3t0d54  ENA
 sd c3t0d55-01   volprdnfs1-01 c3t0d55 0        50630912 294798592 c3t0d55  ENA
 sd c3t0d126-02  volprdnfs1-01 c3t0d126 53024   48247168 345429504 c3t0d126 ENA
 sd c3t0d127-01  volprdnfs1-01 c3t0d127 0       53026176 393676672 c3t0d127 ENA
 sd c3t0d128-01  volprdnfs1-01 c3t0d128 0       53026176 446702848 c3t0d128 ENA
 sd c3t0d129-01  volprdnfs1-01 c3t0d129 0       53026176 499729024 c3t0d129 ENA
 sd c3t0d130-01  volprdnfs1-01 c3t0d130 0       53026176 552755200 c3t0d130 ENA
 sd c3t0d131-01  volprdnfs1-01 c3t0d131 0       53026176 605781376 c3t0d131 ENA
 sd c3t0d132-01  volprdnfs1-01 c3t0d132 0       53026176 658807552 c3t0d132 ENA
 sd c3t0d133-01  volprdnfs1-01 c3t0d133 0       53026176 711833728 c3t0d133 ENA
 sd dgprdnfs04-02 volprdnfs1-01 dgprdnfs04 39544576 13483520 764859904 c3t0d18 ENA
 sd c3t0d55-02   volprdnfs1-01 c3t0d55 50630912 1898016  778343424 c3t0d55  ENA
 sd c3t0d123-03  volprdnfs1-01 c3t0d123 132064  52894112 780241440 c3t0d123 ENA
 sd c3t0d124-03  volprdnfs1-01 c3t0d124 130320  52895856 833135552 c3t0d124 ENA
 sd c3t0d125-02  volprdnfs1-01 c3t0d125 65168   52961008 886031408 c3t0d125 ENA
 sd c3t0d126-03  volprdnfs1-01 c3t0d126 48300192 4725984 938992416 c3t0d126 ENA
 pl volprdnfs1-02 volprdnfs1  ENABLED  TEMPRMSD 943718400 CONCAT   -        WO
 sd c4t2d1-01    volprdnfs1-02 c4t2d1  0        943718400 0        c4t2d1   ENA

 v  volprdnfs2   -            ENABLED  ACTIVE   115343360 SELECT   -        fsgen
 pl volprdnfs2-01 volprdnfs2  ENABLED  ACTIVE   115343360 CONCAT   -        RW
 sd dgprdnfs02-02 volprdnfs2-01 dgprdnfs02 9886464 9886464 0       c3t0d13  ENA
 sd dgprdnfs03-01 volprdnfs2-01 dgprdnfs03 0    53028096 9886464   c3t0d14  ENA
 sd c3t0d203-01  volprdnfs2-01 c3t0d203 0       52428800 62914560  c3t0d203 ENA
 pl volprdnfs2-02 volprdnfs2  ENABLED  TEMPRMSD 115343360 CONCAT   -        WO
 sd c4t2d2-01    volprdnfs2-02 c4t2d2  0        115343360 0        c4t2d2   ENA

 v  volprdnfs3   -            ENABLED  ACTIVE   53026176 SELECT    -        fsgen
 pl volprdnfs3-01 volprdnfs3  ENABLED  ACTIVE   53026176 CONCAT    -        RW
 sd c3t0d122-01  volprdnfs3-01 c3t0d122 0       53026176 0         c3t0d122 ENA
 pl volprdnfs3-02 volprdnfs3  ENABLED  TEMPRMSD 53026176 CONCAT    -        WO
 sd c4t2d3-01    volprdnfs3-02 c4t2d3  0        53026176 0         c4t2d3   ENA

 

Monitor the synchronization process using the vxtask command.

 # vxtask list
 TASKID  PTID TYPE/STATE    PCT   PROGRESS
    161           ATCOPY/R 21.47% 0/53026176/11386880 PLXATT volprdnfs3 volprdnfs3-02 dgprdnfs
    163           ATCOPY/R 05.95% 0/115343360/6862848 PLXATT volprdnfs2 volprdnfs2-02 dgprdnfs
    164           ATCOPY/R 00.52% 0/943718400/4921344 PLXATT volprdnfs1 volprdnfs1-02 dgprdnfs

Once the synchronization process has completed, you are safe to remove the old plexes from the original source copy, or in this case the plexes which contain the c3's subdisks.

 # vxplex -g dgprdnfs -o rm dis volprdnfs1-01
 # vxplex -g dgprdnfs -o rm dis volprdnfs2-01
 # vxplex -g dgprdnfs -o rm dis volprdnfs3-01

Now the mirroring is complete, the following vxprint shows the end result. Disk group dgprdnfs is no longer dependent on controller c3 subdisks and now runs on a new set of storage subdisks c4's .

 # vxprint -qhtg dgprdnfs
 dg dgprdnfs     default      default  16000    1106535203.86.eap42

 dm c4t2d1       c4t2d1s2     auto     2048     964654848 -
 dm c4t2d2       c4t2d2s2     auto     2048     125821696 -
 dm c4t2d3       c4t2d3s2     auto     2048     62909696 -

 v  volprdnfs1   -            ENABLED  ACTIVE   943718400 SELECT   -        fsgen
 pl volprdnfs1-02 volprdnfs1  ENABLED  ACTIVE   943718400 CONCAT   -        RW
 sd c4t2d1-01    volprdnfs1-02 c4t2d1  0        943718400 0        c4t2d1   ENA

 v  volprdnfs2   -            ENABLED  ACTIVE   115343360 SELECT   -        fsgen
 pl volprdnfs2-02 volprdnfs2  ENABLED  ACTIVE   115343360 CONCAT   -        RW
 sd c4t2d2-01    volprdnfs2-02 c4t2d2  0        115343360 0        c4t2d2   ENA

 v  volprdnfs3   -            ENABLED  ACTIVE   53026176 SELECT    -        fsgen
 pl volprdnfs3-02 volprdnfs3  ENABLED  ACTIVE   53026176 CONCAT    -        RW
 sd c4t2d3-01    volprdnfs3-02 c4t2d3  0        53026176 0         c4t2d3   ENA

lsof: Use lsof to Find PID on Port

No need for large scripts to find process ID with Port. If you have the lsof binary available in your system including tr, you can easily find the process ID using the UDP or TCP socket. 

In this example will will locate the process ID using the TCP 1080 port.

 # lsof -i tcp:1080 -Fp | tr -d p

  

veritas: VCS Java Console Logs showing wrong date & timezone

I have a number of Veritas clustered environments that I maintain. I monitor these clusters through the Veritas Cluster Manager JAVA console running on my Windows 7 workstation.

For the last couple of weeks, I have noticed the VCS logs where showing the wrong date and time. Here is a screen shot showing the time I logged in. At the time it was 1 day and almost 12 hours behind.

Beginning in 2007 the U.S.A. and other countries changed the way Daylight Savings Time is scheduled.  The Symantec provided Java Runtime Environments (JREs), VRTSjre,  and VRTSjre15, are affected by the changes in DST. This will cause the time reported internally by the JRE to be off by 1 hour for 4 weeks each year, causing incorrect date and time processing for Symantec applications relying on these JREs. The new DST rules will went into effect in March 2007. To comply with the DST changes, updates to the JREs must be provided.  This is done by either by applying an update to your Symantec product or with a JRE update tool provided by the vendor as described below.
This affects VRTSjre (Java 1.4), VRTSjre15 (Java 1.5),  and 3.2 / 3.3 versions of Veritas Enterprise Administrator (VEA) on all platforms currently supported by these components.

1. To investigate this issue you will need to ensure first if the primary VRTSvcs log from the host is also not showing the wrong date and time:

 Log location: /var/VRTSvcs/log/engine.log_A

2. Also you can double check by running an X Windows session , exporting your DISPLAY variables  and run the hagui locally from the primary host. Then ensure the VCS logs dont also show the wrong date and time.

 # DISPLAY=ip-address:0;export DISPLAY
 # hagui &

By running through the above two basic steps you can confirm which JRE session is having the issue.

Either from your Workstation or the Primary Host.

In my Case the issue was coming from my Windows 7 Workstation.

The following steps will show you how to apply the Java Runtime Environment Timezone Database Update tool to the VRTSjre and Veritas Enterprise Administrator for Daylight Saving Time (DST) changes.

1. You will need to download the latest tzupdater tool to update my JRE timezone running on your workstation.

You should be able to download it from the following link: http://java.sun.com/javase/tzupdater_README.html

2. You must log in using a valid Sun Online account to download.  There is an option to register and create a new account if needed. Log in using your Sun Online account

3. Once you are logged in, agree to the license. This is required to download the tool

4. Click on the Java Standard Edition (SE) download section, and click on the timezone tool download.

5. Once downloaded extract the files to a temporary directory on your workstation, 

6. Update the implementation of the JRE as follows:
For Windows systems, bring up a command prompt session and run the following:

 # cd C:\Program Files\Common Files\VERITAS Shared\VRTSjre\jre1.5\bin\
 java -jar C:\javatz\tzupdater.jar -f -bc -v

To verify:

 # java -jar C:\javatz\tzupdater.jar -t -v

 # cd c:\Program Files\VERITAS\VERITAS Object Bus\jre\bin
 java -jar C:\javatz\tzupdater.jar -f -bc -v

To verify:

 # java -jar C:\javatz\tzupdater.jar -f -bc -v

7. Once you have applied the update to the Java Runtime Environment , you can start a new Veritas Cluster Management JAVA console and ensure the VCS logs have the correct date and time.

8. If you continue to experience issues I suggest to uninstall the Veritas Cluster Management JAVA console, upgrade to the latest version which at time of this post is 5.1.00.2.

5.1 Veritas Cluster Management Console version is backwards compatible. And can be downloaded from the following location:
http://www.symantec.com/business/support/index?page=content&id=TECH78415

JASS: Permanently Disabling Services using JASS

The following will provide some basic instructions on how to disable services on systems using the Solaris Security Toolkit (JASS)

1. Determine what services you wish to disable that you may no longer need. Especially on a WEB farm architecture you would wish to harden/disable uneeded OS services that could potentially be a security threat.

Here is an example of what you can consider.

•  all NFS services (client AND server)
• autofs/automounter
• cde-login and all cde/graphical login components
• drd (only needed for logical domains)
• Sun ServiceTags  (They are usually used to provide discovery of systems and software on the LAN, but I think this is not appropriate for a web host)
• stosreg and sthwreg services.

For example I have disabled: (on top of the usual set of JASS FTP TELNET etc,etc) the following.

CDE and font server

 svc:/application/graphical-login/cde-login

 svc:/network/rpc/cde-calendar-manager

 svc:/network/rpc/cde-ttdbserver

 svc:/application/cde-printinfo

Logical Domains (T series only)

 svc:/platform/sun4v/drd

NFS and automounter

 svc:/network/nfs/cbd  

 svc:/network/nfs/client

 svc:/network/nfs/server

 svc:/network/nfs/mapid

 svc:/network/nfs/status

 svc:/network/nfs/nlockmgr

 svc:/network/nfs/rquota

 svc:/system/filesystem/autofs

Service Tags

 svc:/network/stdiscover:default

 svc:/network/stlisten:default

Removal of the above should leave a nice clean OS with minimal external ports open.

2. To disable such services you will need to use the JASS_SVCS_DISABLE JASS variable in /opt/SUNWjass/Drivers/user.init Use the package-provided file user.init.SAMPLE as a template

 # cd /opt/SUNWjass/Drivers
 # cp user.init.SAMPLE user.init
 # vi user.init

Add the following lines of services to disable.

 JASS_SVCS_DISABLE="${JASS_SVCS_DISABLE}
 svc:/network/telnet:default
 svc:/network/ftp:default
 svc:/application/management/webmin:default
 svc:/system/webconsole:console
 svc:/application/management/snmpdx:default
 svc:/application/management/dmi:default
 svc:/application/management/wbem:default
 svc:/application/graphical-login/cde-login
 svc:/network/rpc/cde-calendar-manager
 svc:/network/rpc/cde-ttdbserver
 svc:/application/cde-printinfo
 svc:/application/font/fc-cache:default
 svc:/platform/sun4v/drd:default
 svc:/network/nfs/cbd
 svc:/network/nfs/client
 svc:/network/nfs/server
 svc:/network/nfs/mapid
 svc:/network/nfs/status
 svc:/network/nfs/nlockmgr
 svc:/network/nfs/rquota
 svc:/system/filesystem/autofs
 svc:/network/stdiscover:default
 svc:/network/stlisten:default "
 export JASS_SVCS_DISABLE

3. To apply the changes run jass-execute in apply mode again and reboot:

 # /opt/SUNWjass/bin/jass-execute -d server-secure.driver

JASS: Installing the Solaris Security Toolkit JASS

The Solaris Security Toolkit provides a flexible way to harden a Solaris system, making it more secure from malicious attack. This software may be installed during a unattended Solaris JumpStart install, or installed after Solaris is installed and booted. 

The degree of hardening depends on your architectural requirements. I've been happily using JASS for the last year now, I have my own specific JASS build packages for various environments, I also run audits on a weekly basis to ensure there have been no malicious system or software changes. 

Pre-canned scripts provide various levels of hardening & auditing. 
The server-secure.driver leaves frequently-used server services open. The following discusses customizing the server-secure.driver to your site-specific needs. Once customized, your systems can be hardened in an automated way using one or more configurations you established.

Here I'll give enough details to get you started. I'm only covering interactive use in this example.

1. First, download the SUNWjass 4.2.0 package, available at http://www.sun.com/software/security/jass/. You don't need the MD5 (SUNBEmd5) or fixmodes (SUNBEfixm) packages, as the functionality is incorporated in Solaris 10. 

2. If SUNWjass is already installed, remove it with pkgrm (back up any modified files first) 

3. Uncompress and install the package to /opt/SUNWjass

 # uncompress SUNWjass.pkg.Z
 # pkgadd -d SUNWjass.pkg

4. Run a driver in "apply" mode. In this example, we use server-secure.driver This takes a few minutes.

 # /opt/SUNWjass/bin/jass-execute -d server-secure.driver

5. Check the summary output for failures and errors:

 [SUMMARY] Results Summary for APPLY run of server-secure.driver
 [SUMMARY] The run completed with a total of 85 scripts run.
 [SUMMARY] There were  Failures in   0 Scripts
 [SUMMARY] There were  Errors   in   0 Scripts
 [SUMMARY] There were  Warnings in   2 Scripts
 [SUMMARY] There were  Notes    in  68 Scripts

6. Reboot and login again:

7. You can verify the previous run of jass-execute and manually run audit of the system

 # /opt/SUNWjass/bin/jass-execute -a server-secure.driver

This takes a few minutes and produces a summary at the end:

 [SUMMARY] Results Summary for AUDIT run of server-secure.driver
 [SUMMARY] The run completed with a total of 85 scripts run.
 [SUMMARY] There was a Failure  in   1 Script
 [SUMMARY] There were  Errors   in   0 Scripts
 [SUMMARY] There was a Warning  in   1 Script
 [SUMMARY] There were  Notes    in  20 Scripts

 [SUMMARY] Failure Scripts listed in:
         /var/opt/SUNWjass/run/20050721092746/jass-script-failures.txt

Verify there are no failures. If any failures are found as in this case, look at the script output to see if there are any unexpected problems. In the example above, I see the failure is from set-root-home-dir.aud because I provided and created a custom .profile script: 

 [FAIL] Template /root/.profile does not match target on system.

You can fix it by removing my custom .profile file, or modifying the .profile in Solaris Security Toolkit (under /opt/SUNWjass/Files).I prefer the later.
Change the root .profile to your desire from the following location  /opt/SUNWjass/Files/root/.profile
Then run another hardening:

 # /opt/SUNWjass/bin/jass-execute -d server-secure.driver

Then run another audit to confirm there are no errors this time.

 # /opt/SUNWjass/bin/jass-execute -a server-secure.driver

 [SUMMARY] Results Summary for AUDIT run of server-secure.driver
 [SUMMARY] The run completed with a total of 85 scripts run.
 [SUMMARY] There was a Failure  in   0 Script
 [SUMMARY] There were  Errors   in   0 Scripts
 [SUMMARY] There was a Warning  in   1 Script
 [SUMMARY] There were  Notes    in  20 Scripts