controlling cpu usage part 2: CPU Usage Limit in the Shell

The shell ulimit command can be used to check or set the CPU limit for any subsequently created children, and their descendants. The -t option to ulimit sets the amount of CPU time a process may use before it is sent a SIGXCPU signal by the kernel. The default  is unlimited (maximum CPU time).

 # su - useruser $ ulimit -t
 unlimited
 user $ sh
 user $ ulimit -t
 unlimited
 user $ ulimit -t 10
 user $ date; while : ; do : ; done; date
 Friday, 5 September 2008  3:34:56 PM EST
 Cpu Limit Exceeded  (core dumped)
 user $

controlling cpu usage part 1: Introduction

CPU usage can be controlled in a number of different ways. The possible choices as of Solaris 05/08 are:

• We can set a CPU usage limit in the shell
• We can manipulate the dispatch parameter kernel tables
• We can use different schedulers, such as the FSS (Fair Share Scheduler)
• We can bind a process to a CPU
• We can use processor sets
• We can create pools, which combine scheduler changes and processor sets
• We can set a capped-cpu resource control for solaris container or zones

In the coming weeks I will discuss these options in a little detail in hope you can improve performance or tune aspects of your environment better.

solaris: Description of all services

A quick tip for all Solaris 10/OpenSolaris users… some companies have a strict requirement to know exactly what each and every startup script does on their system. With releases of Solaris 9 and earlier, one would check the rc scripts. This is time consuming and may not give an accurate description or one liner. Solaris 10/OpenSolaris makes things much easier…

Solaris 10/OpenSolaris now uses services for all the Sun supplied start up scripts, but still supports the old (legacy) rc scripts too. A summary of all services can be obtained by running: svcs -o FMRI,DESC. This will produce output similar to the following:

 #svcs -o FMRI,DESC
  FMRI                                               DESC
  lrc:/etc/rc2_d/S00set-tmp-permissions              -
  lrc:/etc/rc2_d/S07set-tmp-permissions              -
  lrc:/etc/rc2_d/S10lu                               -
  lrc:/etc/rc2_d/S20sysetup                          -
  lrc:/etc/rc2_d/S40llc2                             -
  lrc:/etc/rc2_d/S42ncakmod                          -
  lrc:/etc/rc2_d/S70nddconfig                        -
  lrc:/etc/rc2_d/S72autoinstall                      -
  lrc:/etc/rc2_d/S73cachefs_daemon                   -
  lrc:/etc/rc2_d/S81dodatadm_udaplt                  -
  lrc:/etc/rc2_d/S89bdconfig                         -
  lrc:/etc/rc2_d/S91afbinit                          -
  lrc:/etc/rc2_d/S91gfbinit                          -
  lrc:/etc/rc2_d/S91ifbinit                          -
  lrc:/etc/rc2_d/S91jfbinit                          -
  lrc:/etc/rc2_d/S91kfbinit                          -
  lrc:/etc/rc2_d/S91zuluinit                         -
  lrc:/etc/rc2_d/S94ncalogd                          -
  lrc:/etc/rc2_d/S95lwact                            -
  lrc:/etc/rc2_d/S95nbclient                         -
  lrc:/etc/rc2_d/S98deallocate                       -
  lrc:/etc/rc2_d/S99sneep                            -
  lrc:/etc/rc3_d/S16boot_server                      -
  lrc:/etc/rc3_d/S50apache                           -
  lrc:/etc/rc3_d/S52imq                              -
  lrc:/etc/rc3_d/S84appserv                          -
  svc:/system/fpsd:default                           FP Scrubber - Online Floating Point Unit Test
  svc:/system/svc/restarter:default                  master restarter
  svc:/network/pfil:default                          packet filter
  svc:/network/tnctl:default                         trusted networking templates
  svc:/network/loopback:default                      loopback network interface
  svc:/system/installupdates:default                 system update installer
  svc:/system/filesystem/root:default                root file system mount
  svc:/system/scheduler:default                      default scheduling class configuration
  svc:/system/boot-archive:default                   check boot archive content
  svc:/network/physical:default                      physical network interfaces
  svc:/system/identity:node                          system identity (nodename)
  svc:/system/filesystem/usr:default                 read/write root file systems mounts
  svc:/system/keymap:default                         keyboard defaults
  svc:/network/ipfilter:default                      IP Filter
  svc:/system/device/local:default                   Standard Solaris device configuration.
  svc:/system/filesystem/minimal:default             minimal file system mounts
  svc:/system/rmtmpfiles:default                     remove temporary files
  svc:/system/resource-mgmt:default                  Global zone resource management settings
  svc:/system/coreadm:default                        system-wide core file configuration
  svc:/system/name-service-cache:default             name service cache
  svc:/system/identity:domain                        system identity (domainname)
  svc:/system/cryptosvc:default                      cryptographic services
  svc:/system/sysevent:default                       system event notification
  svc:/system/device/fc-fabric:default               Solaris FC fabric device configuration.
  svc:/network/ipsec/ipsecalgs:default               IPsec algorithm initialization
  svc:/milestone/devices:default                     device configuration milestone
  svc:/system/picl:default                           platform information and control
  svc:/network/ipsec/policy:default                  IPsec policy initialization
  svc:/milestone/network:default                     Network milestone
  svc:/system/pkgserv:default                        Flush package command database to disk
  svc:/application/print/ppd-cache-update:default    ppd cache update
  svc:/network/initial:default                       initial network services
  svc:/system/manifest-import:default                service manifest import
  svc:/network/service:default                       layered network services
  svc:/system/patchchk:default                       Launcher for Automatic Patching services
  svc:/network/dns/client:default                    DNS resolver
  svc:/milestone/name-services:default               name services milestone
  svc:/network/iscsi/initiator:default               -
  svc:/milestone/single-user:default                 single-user milestone
  svc:/platform/sun4v/efdaemon:default               embedded FCode interpreter
  svc:/system/filesystem/local:default               local file system mounts
  svc:/network/shares/group:default                  Share Group
  svc:/system/cron:default                           clock daemon (cron)
  svc:/network/shares/group:zfs                      Share Group
  svc:/system/sysidtool:net                          sysidtool
  svc:/system/boot-archive-update:default            update boot archive if necessary
  svc:/network/routing-setup:default                 Initial routing-related configuration.
  svc:/network/ntp:default                           Network Time Protocol (NTP)
  svc:/network/rpc/bind:default                      RPC bindings
  svc:/application/psncollector:default              Product Serial Number Collector
  svc:/system/sysidtool:system                       sysidtool
  svc:/milestone/sysconfig:default                   Basic system configuration milestone
  svc:/system/sac:default                            SAF service access controller
  svc:/system/postrun:default                        Postponed package postinstall command execution
  svc:/network/inetd:default                         inetd
  svc:/system/utmp:default                           utmpx monitoring
  svc:/system/console-login:default                  Console login
  svc:/system/dumpadm:default                        system crash dump configuration
  svc:/network/ssh:default                           SSH server
  svc:/system/system-log:default                     system log
  svc:/application/management/seaport:default        net-snmp SNMP daemon
  svc:/network/smtp:sendmail                         sendmail SMTP mail transfer agent
  svc:/network/sendmail-client:default               sendmail SMTP client queue runner
  svc:/system/fmd:default                            Solaris Fault Manager
  svc:/network/rpc/rstat:default                     kernel statistics server
  svc:/network/rpc/smserver:default                  removable media management
  svc:/network/cde-spc:default                       CDE subprocess control
  svc:/network/bpcd/tcp:default                      bpcd
  svc:/network/vnetd/tcp:default                     vnetd
  svc:/network/vopied/tcp:default                    vopied
  svc:/network/bpjava-msvc/tcp:default               bpjava-msvc
  svc:/system/filesystem/volfs:default               Volume Management filesystem
  svc:/application/management/sma:default            net-snmp SNMP daemon
  svc:/milestone/multi-user:default                  multi-user milestone
  svc:/milestone/multi-user-server:default           multi-user plus exports milestone
  svc:/application/stosreg:default                   Service Tag OS Registry Inserter
  svc:/system/zones:default                          Zones autoboot and graceful shutdown
  svc:/system/basicreg:default                     
  svc:/application/sthwreg:default                   Hardware Service Tag Collector
  svc:/application/print/ipp-listener:default        Internet Print Protocol Listening Service
  svc:/application/print/rfc1179:default             BSD print protocol adapter

veritas: Cannot unmount a Locked VxFS filesystem

Storage Foundation 5.0MP3 introduced a new feature called VxFS filesystem lock which disallows accidental unmounts when the  file system resource is online. New umount option mntunlock is used to clear the lock and then unmount the filesystem. The offline script for the Mount resource will use this new option.

How to check if the filesystem is locked by VCS:

 # mount -v | grep mntlock

Sometimes, it may be necessary to unmount a mount locked filesystem. This is for cases where VCS service groups having DiskGroup resources configured with UnMountVolumes attribute set and the volumes are mounted outside of VCS control (this is not very common).

The filesystem locking system is to prevent accidental unmounts, if the attribute is set to 0 VCS will not lock the filesystem.
















The following fix from Symantec to umount the locked vxfs filesystem may not work, a bug in 5.0 MP3 was found and it could be fixed in future releases. 
If the following command does not work a bounce is required.

Solaris:

 # /opt/VRTS/bin/umount -o mntunlock=VCS /mount-point

If you continue to experience issues such as :

 UX:vxfs umount: ERROR: V-3-21705: mount-point cannot unmount : Device busy

The above "umount" command has already cleared the Mount Lock silently but the Mount Lock is still shown in the "mount -v" output.   Now the file system will not be able to be unmounted.
Run fuser checks on the mount points to confirm any outbound processes still running:

 # fuser -c /mount-point

but if you continue to encounter issues and recieve the following error:

 UX:vxfs umount: ERROR: V-3-26365: Incorrect mntlock id  (Invalid argument)

You can attempt to unmount it using the fsadm command:

 # fsadm -o mntunlock=VCS /mount-point

Trying to clear the Mount Lock using fsadm could also fail.

 UX:vxfs fsadm: ERROR: V-3-26348: file system not mount locked

The workaround is to lock the file system again using fsadm with the same lock name.

 # fsadm -o mntlock=VCS /mount-point

Now the system can be unmounted successfully by umount.

 # umount -o mntunlock=VCS /mount-point

Please note that if the VxFS file system is disabled, fsadm will not be able to remove the lock.   The only way to unmount the disabled file system is to reboot the system.

 Apr 17 13:00:07 alaw2 vxfs: [ID 702911 kern.warning] WARNING: msgcnt 3 mesg 031: V-2-31: vx_disable  - /mount-point file system disabled

 # fsadm -o mntlock=VCS /mount-point
 UX:vxfs fsadm: ERROR: V-3-20275: cannot open /mount-point

veritas: Mult-Link-based IPMP setup with VCS

With Solaris 10 came a nice feature – Multi-Link-based IP Multipathing (IPMP). It determines NIC availability solely on the NIC driver reporting the physical link status – UP or DOWN. Previous versions used “probe-based” IPMP, where connectivity is tested by pinging something on the network from each interface. While probe-based is actually a more thorough test (tests network layer 3 as well as 2), it is much more cumbersome to configure, and you need an extra IP address for each interface for “test” addresses. IMO Multi-Link-based IPMP is sufficient for most applications.
To achieve multi-link-based IPMP, here’s how I’ve configured my MultiNICB resource in this large 10 node clustered environment:

Multi-Link-based IPMP MultiNICB properties

These are the values you must change from the defaults:
UseMpathd: 1
Tells VCS to use mpathd for network link status
MpathCommand: /usr/sbin/in.mpathd
Be sure to create a symbolic link to /usr/lib/inet/in.mpathd -a if the above does not exist. 
ConfigCheck: 1
If you leave this at 1, it will overwrite your /etc/hostname.xxx files with probe-based IPMP configuration, if left at 0 it will not change.
Device: (your IPMP interfaces here)
List of interfaces and there interface aliases.
Tick on per System and add the device and interface alias entry for each IPMP grouped interface from each host in the cluster.
GroupName:
Do not use your IPMP group name here, it’s not needed. VCS is not monitoring the group, mpathd is.

veritas: Veritas Volume Manager (VxVM) Commands

Here are some links to Basic and Advanced (VxVM) commands for your online storage management enterprise or Storage Area Network (SAN) environments.

Basic VxVm Commands:
Advance VxVM Commands:

commands: eXtended System Control Facility (XSCF)

The Sun SPARC Enterprise M4000/M5000/M8000/M9000 Servers XSCF useful console commands:

 XSCF> console -d 0
 XSCF> console -f -d 0
 XSCF> showstatus
 XSCF> showversion -c xcp -v [shows xcp firmware, version, openboot prom version
 XSCF> showenvironment
 XSCF> showenvironment temp
 XSCF> showenvironment volt
 XSCF> showhardconf
 XSCF> showdcl -va [check domain id...]
 XSCF> showdomainstatus -a
 XSCF> showboards -a
 XSCF> poweron -a [powers up all domains]
 XSCF> poweroff -a [powers off all domains]
 XSCF> poweron -d 0 [powers on domain 0]
 XSCF> poweroff -d 0 [powers off domain 0]
 XSCF> poweroff -f -d 0 [forces a power off domain 0]
 XSCF> reset -d 0 por [resets domain 0]
 XSCF> reset -d 0 xir [resets domain 0 with XIR reset]
 XSCF> sendbreak -d 0 [sends break command to domain 0]
 XSCF> setautologout -s 60 [sets autologout to 60 minutes]
 XSCF> showautologout
 XSCF> shownetwork -a
 XSCF> setnetwork xscf#0-lan#0 -m 255.255.255.0 10.10.10.5
 XSCF> sethostname xscf#0 fire-xscf
 XSCF> sethostname -h host.org
 XSCF> setroute -h host.org
 XSCF> setnameserver 10.10.10.2 10.10.10.3
 XSCF> setroute -c add -n 10.10.10.1 -m 255.255.255.0 xscf#0-lan#
 XSCF> snapshot -L F -t [username]@[hostname]:[directory_to_save_to]


 #. to break from console