java: Alert on java crashes

The last couple of days I have noticed a number of irregular core dumps created in my system core file dump location /var/core

 -rw-------   1 root    root  2529790203 Nov 10 11:55 core_host1_java_1094_300_1289350401_28578
 -rw-------   1 root    root  2564932547 Nov 15 13:06 core_host1_java_1094_300_1289786684_1664
 -rw-------   1 root    root  2498732827 Nov 17 17:29 core_host1_java_9092_300_1289975232_5664
 -rw-------   1 root    root  2525420387 Nov 19 12:08 core_host3_java_1094_300_1290128885_16234

Depending on how you've setup your core file dump pattern, you can determine which process/application user account its comming from by just reading the file core name. eg

 # coreadm|grep pattern
    global core file pattern: /var/core/core_%n_%f_%u_%g_%t_%p
    init core file pattern: /var/core/core_%n_%f_%u_%g_%t_%p

 %n ; system node name uname -n
 %f ; executable filename
 %u ; uid
 %g ; gid
 %t ; time in seconds since 1970,1,1.
 %p ; PID

 
My core dump process is coming from a java process. Bugs can occur in a Java runtime environment and most administrators would want to get notified.
If you need to take a corrective action and diagnose further, you will need to be alerted at the time of incident.
The Java runtime has a number of useful options that can be used for this purpose. The first option is “-XX:OnOutOfMemoryError”, which allows a command to be run when the runtime environment incurs an out of memory condition. When this option is combined with the logger command line utility:

 java -XX:OnOutOfMemoryError=”logger Java process %p encountered an OOM condition” …

Syslog entries will be generated each time an Out Of memory (OOM) event occurs.

Another useful option is “-XX:OnError”, which allows a command to be run when the runtime environment incurs a fatal error (i.e., a hard crash). When this option is combined with the logger utility:

 java -XX:OnError=”logger -p Java process %p encountered a fatal condition” …

Syslog entries will be generated when a fatal event occur.

The options above allow you to run one or more commands when these errors are encountered, so you could chain together a utility (logger or mail) to generate alerts, and maybe a restarter script to start a new Java process.

veritas: VCS Java Console Logs showing wrong date & timezone

I have a number of Veritas clustered environments that I maintain. I monitor these clusters through the Veritas Cluster Manager JAVA console running on my Windows 7 workstation.

For the last couple of weeks, I have noticed the VCS logs where showing the wrong date and time. Here is a screen shot showing the time I logged in. At the time it was 1 day and almost 12 hours behind.

Beginning in 2007 the U.S.A. and other countries changed the way Daylight Savings Time is scheduled.  The Symantec provided Java Runtime Environments (JREs), VRTSjre,  and VRTSjre15, are affected by the changes in DST. This will cause the time reported internally by the JRE to be off by 1 hour for 4 weeks each year, causing incorrect date and time processing for Symantec applications relying on these JREs. The new DST rules will went into effect in March 2007. To comply with the DST changes, updates to the JREs must be provided.  This is done by either by applying an update to your Symantec product or with a JRE update tool provided by the vendor as described below.
This affects VRTSjre (Java 1.4), VRTSjre15 (Java 1.5),  and 3.2 / 3.3 versions of Veritas Enterprise Administrator (VEA) on all platforms currently supported by these components.

1. To investigate this issue you will need to ensure first if the primary VRTSvcs log from the host is also not showing the wrong date and time:

 Log location: /var/VRTSvcs/log/engine.log_A

2. Also you can double check by running an X Windows session , exporting your DISPLAY variables  and run the hagui locally from the primary host. Then ensure the VCS logs dont also show the wrong date and time.

 # DISPLAY=ip-address:0;export DISPLAY
 # hagui &

By running through the above two basic steps you can confirm which JRE session is having the issue.

Either from your Workstation or the Primary Host.

In my Case the issue was coming from my Windows 7 Workstation.

The following steps will show you how to apply the Java Runtime Environment Timezone Database Update tool to the VRTSjre and Veritas Enterprise Administrator for Daylight Saving Time (DST) changes.

1. You will need to download the latest tzupdater tool to update my JRE timezone running on your workstation.

You should be able to download it from the following link: http://java.sun.com/javase/tzupdater_README.html

2. You must log in using a valid Sun Online account to download.  There is an option to register and create a new account if needed. Log in using your Sun Online account

3. Once you are logged in, agree to the license. This is required to download the tool

4. Click on the Java Standard Edition (SE) download section, and click on the timezone tool download.

5. Once downloaded extract the files to a temporary directory on your workstation, 

6. Update the implementation of the JRE as follows:
For Windows systems, bring up a command prompt session and run the following:

 # cd C:\Program Files\Common Files\VERITAS Shared\VRTSjre\jre1.5\bin\
 java -jar C:\javatz\tzupdater.jar -f -bc -v

To verify:

 # java -jar C:\javatz\tzupdater.jar -t -v

 # cd c:\Program Files\VERITAS\VERITAS Object Bus\jre\bin
 java -jar C:\javatz\tzupdater.jar -f -bc -v

To verify:

 # java -jar C:\javatz\tzupdater.jar -f -bc -v

7. Once you have applied the update to the Java Runtime Environment , you can start a new Veritas Cluster Management JAVA console and ensure the VCS logs have the correct date and time.

8. If you continue to experience issues I suggest to uninstall the Veritas Cluster Management JAVA console, upgrade to the latest version which at time of this post is 5.1.00.2.

5.1 Veritas Cluster Management Console version is backwards compatible. And can be downloaded from the following location:
http://www.symantec.com/business/support/index?page=content&id=TECH78415