zones: Physical to Virtual (P2V) Migration

Since the release of Solaris 10 Update 9 I was interested in trying out its new capabilities. One of which was to move from an existing Oracle Solaris 10 Physical system to a virtual container quickly and easily on a separate system. And also the Host ID migration feature which it also allows.

In this post I will provide a demonstration of P2V , migrating a physical Solaris system into a zone, using Host ID migration, keeping in mind in this example both systems are built from a Solaris 10 Update 9 release and is a ZFS on root build.





1. Collect the system information you wish to be migrated into a zone.
Obtain the hostname:

 # hostname

Obtain the hostid:

 # hostid

 
Obtain the root passwd
View what software being run on the system if necessary
Check the network configuration on the system:

 # ifconfig -a   

View the storage utilized for example the contents of /etc/vfstab
View the amount of local disk storage in use, and determine the target host has enough space for the install
Examine the /etc/system of any specific or unique changes you may wish to keep or change once the  environment has been virtualized.

2. Use the flarcreate command to create a flash archive image of the system you wish to migrate.
Login as root to the source system and change to the root directory.

 # cd /

   
Run the flarcreate using the following options:
Note: Here I have used the option to compress using the -c , used the archived method with cpio, you can use pax if you wish, then supplied the content name identifier of the archive using the -n option (best practice s10u9-system-name), also I have used the -y option because this system has a separate /var dataset and is ZFS on root based, then supplied the path destination of the flar archive creation.
When the flar creation is running be sure to monitor and review any errors thoroughly.

 # flarcreate -c -L cpio -n name -y /var /path/to/flar

3. Transfer the flar archive created to the destination target host.

4. Create a new zone configuration on the target host:
Note: I don't want to inherit any packages from the global or any loop-back file systems, so I have created a whole root zone configuration and provided a new  ip-address as I don't want it to conflict with the existing migrated host as its still alive.

Also you will see I have added a hostid entry in the zone configuration, when applications are migrated from a physical Solaris system into a zone on a new system, the hostid changes to be the hostid of the new machine. In some cases, applications depend on the original hostid, and it is not possible to update the application configuration. In these cases, the zone can be configured to use the hostid of the original system. This is done by setting a zonecfg property to specify the hostid as shown below. The value used should be the output of the hostid command as run on the original system previously.

 # zonecfg -z hostname
 hostname: No such zone configured
 Use 'create' to begin configuring a new zone.
 zonecfg:hostname> create -b
 zonecfg:hostname> set autoboot=true
 zonecfg:hostname> set zonepath=/zones/hostname
 zonecfg:hostname> set bootargs="-m verbose"
 zonecfg:hostname> set hostid=84###375
 zonecfg:hostname> add net
 zonecfg:hostname:net> set physical=bge0
 zonecfg:hostname:net> set address=ip-address
 zonecfg:hostname:net> end
 zonecfg:hostname> verify
 zonecfg:hostname> commit
 zonecfg:hostname> exit

5. Installing the zone on the target system using the flar archive created. Become root user and install the configured zone using the install -a option and the path to the flar archive created. Noticed I have used the -p option becuase I want to preserve the system identity, the zone will have the same identity as the system used to create the image. You can use the -u to sys-unconfig the zone.
Best practice make sure you tail the zone installation log file and ensure no errors are found.

 # zoneadm -z hostname install -p -a /path/to/flar
 A ZFS file system has been created for this zone.
       Log File: /var/tmp/hostname.install_log.IQaGnI
     Installing: This may take several minutes...
 Postprocessing: This may take a while...
    Postprocess: Updating the zone software to match the global zone...
    Postprocess: Zone software update complete
    Postprocess: Updating the image to run within a zone

         Result: Installation completed successfully.
       Log File: /zones/hostname/root/var/log/hostname.install17462.log

6. Boot the zone into single user mode and login via the console and of course use the root password from your migrated host. Make any necessary checks.

 # zoneadm -z hostname boot -s
 # zlogin -C hostname

From here you can see the P2V is complete, hostname and data has been kept intact including the host-id.
Now you can decide whether you need to make any further necessary changes such as hostname, network configuration etc etc. And of course be prepared to boot the migrated zone for a live production environment.

controlling cpu usage part 9: Setting capped-cpu for a Zone

Introduced in Solaris 10 05/08 CPU caps allow a fine division of CPU resources. The administrator can allocate CPU resources in 1% increments of a single CPU. The allocation can go from 1% of a single CPU to the of CPU's in the system.

The capped-cpu resource type has a single property ncpus. This holds the amount of CPU allocation for the zone. It is expressed in units of a CPU, so 0.01 would be 1% of a single CPU, 1 would be 1CPU and 3.75 would be 3 3/4 CPU's.

I there are multiple CPU's in the system the allocation can come from any CPU, so multi-threaded code can still run threads in parallel if the scheduler so allocates.

However, unlike pools there is no dynamic balancing. If capped-cpu is enabled the CPU resources are statically divided. Unused CPU cycles in a zone are not available for other zones which have capped-cpu in effect.

To set a zone to use 18% of a single CPU we would enter the following.

 # zonecfg -z test0z1
 zonecfg:test0z1> add capped-cpu
 zonecfg:test0z1:capped-cpu> set ncpus=0.18
 zonecfg:test0z1:capped-cpu> end
 zonecfg:test0z1> exit

One important point to remember is that a single threaded process cannot utilize more than a single CPU irrespective of the value of the value of the capped-cpu resource.

You can check to see the performance of the capped-cpu zone using the prstat -Z command.

The percentage (of the global CPU resource) utilized by each zone will be listed with each zone in its summary line.

controlling cpu usage part 8: Adding Pools to a Zone

Zones are able to use the pool subsystem directly. When a zone is defined it can be associated with a named pool by setting the zone's pool property to the name of an existing pool.

 # zonecfg -z zone set pool=pool_web

Multiple zones may share the same pool. In this case each zone should set the cpu-shares resource type to arbitrate between the relative use of CPU for each zone in the pool.

 # zonecfg -z test0z1 set cpu-shares=20
 # zonecfg -z test0z2 set cpu-shares=30

Solaris 11/06 introduces the concept of anoymous pools. These are pools created by a zone when it boots for the exclusive use of that zone. This is done through the dedicated-cpu resource type for a zone. The dedicated-cpu resource type has two properties, ncpus which indicates the number CPU's to put into the created pool, or a range of CPU's if a dynamic pool is desired, and importance which sets the pool.importance property in the pool for use as tie-breaker by poold.

 # zonecfg -z test0z1
 zonecfg:test0z1> add dedicated-cpu
 zonecfg:test0z1:dedicated-cpu> set ncpu=1-3
 zonecfg:test0z1:dedicated-cpu> set importance=10
 zonecfg:test0z1:dedicated-cpu> end
 zonecfg:test0z1> commit
 zonecfg:test0z1> exit

Whenever the zone boots the zoneadmd deamon will create a pool and assign the zone to the pool. 

Note that the dedicated-cpu resource on a zone means that the pool cannot be shared between multiple zones.

If the FSS is desired for the zone then the scheduling-class resource type for the zone should also be set.

 # zonecfg -z test0z1
 zonecfg:test0z1> set scheduling-class=FSS
 zonecfg:test0z1> commit
 zonecfg:test0z1> exit

Note The pools system must be already configured on the system before the dedicated-cpu resource type is used by the zone. If the pool system is not configured any attempt to boot the pool will result in an error from zoneadm.

If there is not enough resources to create the pool an attempt to boot results in a fatal error, and the boot fails.

 # zoneadm -z test0z1 boot
 zoneadm: zone 'test0z1': libpool(3LIB) error: invalid configuration
 zoneadm: zone 'test0z1': dedicated-cpu setting cannot be instatiated
 zoneadm: zone 'test0z1': call to zoneadmd failed

When the zone is booted a temporary pool called SUNWtmp_zonename is created.

        pool SUNWtmp_test0z1
                int     pool.sys_id 4
                boolean pool.active true
                boolean pool.default false
                int     pool.importance 10
                string  pool.comment
                boolean pool.temporary true
                pset    SUNWtmp_test0z1

        pset SUNWtmp_test0z1
                int     pset.sys_id 1
                boolean pset.defaul false
                uint    pset.min 1
                uint    pset.max 3
                string  pset.units population
                uint    pset.load 1991
                uint    pset.size 1
                string  pset.comment
                boolean pset.temporary true

                cpu
                        int     cpu.sys_id 0
                        string  cpu.comment
                        string  cpu.status on-line

The dedicated-cpu resource type creates a pool for the exclusive use of this zone. The zone has exclusive access to the CPU's in the pool. For that reason the cpu-shares resource type in the zone has no meaning if a dedicated-cpu resource type is also defined. The zone will always have 100% of the shares in the processor set, and so will always have the entire processor set to itself irrespective of the number of shares. 

controlling cpu usage part 4: The Fair Share Scheduler

The Fair Share Scheduler (FSS) is an alternative scheduling class. It is not used by default an is explicitly enabled. The FSS guarantees a minimum proportion of the machines CPU resources are made available to each holder of shares, in proportion of the number of shares held.

The absolute quantity of shares is not important. Any number that is in proportion with the desired CPU entitlement can be used.

To configure projects the /etc/project file needs to be modified to identify the number of shares to be granted to each project, and the /etc/user_attr file needs to be modified to assign each user to a project.

To define two users, u1 and u2 with u1 having twice the CPU resources as u2 the entries in /etc/user_attr and /etc/project would be similar to the following:

 # egrep 'u[12]' /etc/passwd
 u1:x:1000:1::/export/home/u1:/bin/sh
 u2:x:1001:1::/export/home/u2:/bin/sh

 # egrep 'u[12]' /etc/user_attr
 u1::::type=normal;project=u1
 u2::::type=normal;project=u2

 # egrep 'u[12]' /etc/project
 u1:1000:User 1:u1::project.cpu-shares=(privileged,20,none)
 u2:1001:User 2:u2::project.cpu-shares=(privileged,10,none)

To determine the project of the current process the ps command may be used, and the prctl command will show the number of shares.

 # ps -o project= -p $$
 user.root

 # su - u1
 $ ps -o project= -p $$
          u1

 $ prctl -t privileged -n project.cpu-shares -i pid $$
 process: 1444: -sh
 NAME      PRIVILEGED         VALUE       FLAG    ACTION          RECIPIENT
 project.cpu-shares
           privileged            20                               None            -         

To change the scheduling class of a running process you can use the priocntl command.

 #  priocntl -s -c FSS -i pid            # Change one process
 #  priocntl -s -c FSS -i class TS       # Change everything currently in TS
 #  priocntl -s -c FSS -i zoneid 1       # Change all processes in zone ID 1
 #  priocntl -s -c FSS -i pid 1          # Change init (special case)

To examine the shares granted to a process (or zone) use the prctl command.

 # prctl -t privileged -n zone.cpu-shares -i zoneid 1     # Shares for zone ID 1

To modify the number of shares granted to a zone we can use -r option to prctl. This change only lasts until next reboot.

 # prctl -r -v 10 -t privileged -n zone.cpu-shares -i zoneid 1  
 # Change number of shares to 10

To change the default scheduling class, so that on next and subsequent reboots all process will use FSS by default we can use the dispadmin command.

 # dispadmin -d FSS

zones: TTYsrch Issues Within Zones

Symptoms such as process-related commands like "ps", "ps -ef", "ptree", "pgrep" etc. were hanging and not completing. This had knock-on effects for utilities and applications which use the ps command, such as VCS, VxVM etc. 

The above problem only occurred when one or more non-global zones was running on the system.

Root Cause:

Among other things, the ps command traverses the system "device tree" in /dev, to link processes with terminal devices. Such searches are optimized using the /etc/ttysrch file, which is like a PATH file for the /dev directory. If the required device is not found under one of the directories listed in the /etc/ttysrch file, it must do a recursive search of the whole /dev directory, which can take a long time. This is what was happening in the above case.

When non-global zones are running, the relevant terminal device files reside under the /dev/zcons directory. This entry was missing from the /etc/ttysrch file. It seems that servers installed with Solaris 10 to start with have this entry included in their ttysrch file, but on servers upgraded from Solaris 8 to 10 via Live Upgrade, this entry is missing. This would appear to be a bug in the Sun Live Upgrade process.

Fix:
 

Add the entry "/dev/zcons" to the /etc/ttysrch file. See the following examples - last few lines of /etc/ttysrch in each case.

 #
 /dev/pts
 /dev/term
 /dev/xt
 /dev/zcons