Solaris tips and tricks - wiki

Skip to end of metadata

Go to start of metadata

Solaris 10

Creating a new Zone

bash-3.00# zonecfg -z zone1
imsb-lims: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:zone1> create
zonecfg:zone1> set zonepath=/home/zones/zone1
zonecfg:zone1> set autoboot=false
zonecfg:zone1> add net
zonecfg:zone1:net> 
zonecfg:zone1:net> 
zonecfg:zone1:net> 
zonecfg:zone1:net> set address=129.132.230.223
zonecfg:zone1:net> set physical=aggr2
zonecfg:zone1:net> end
zonecfg:zone1> add fs
zonecfg:zone1:fs> set dir=/data
zonecfg:zone1:fs> set special=/data/openbis
zonecfg:zone1:fs> set type=lofs
zonecfg:zone1:fs> add options [nodevices]
zonecfg:zone1:fs> end
zonecfg:zone1> exit

bash-3.00# zoneadm -z zone1 verify
bash-3.00# zoneadm -z zone1 install

bash-3.00# zoneadm -z zone1 boot

bash-3.00# zlogin -e \; -C zone1          # The -C option to zlogin specifies the target zone. The -e option to zlogin specifies the escape character.
Ie: to escape - do ;.

After login, complete the normal Solaris install. (Answer the questions)

To create a read/write sudirectory in a read-only filesystem:

Example: In order that the zones can write to /opt/oracle/9.2.0/dbs

In zone

mkdir /oracledbs

mount -F lofs /oracledbs /opt/oracle/9.2.0/dbs

Put the following entry in /etc/vfstab:

/oracledbs      -       /opt/oracle/9.2.0/dbs   lofs    -       yes     -

In order that a zone can write to an existing directory.
Example: being able to change config files in /opt/oracle/9.2.0/Apache

In global zone

mv /opt/oracle/9.2.0/Apache /opt/oracle/9.2.0/xApache

mkdir /opt/oracle/9.2.0/Apache  # make a mountpoint only

In zone

mkdir /oraApache

mount -F lofs /oraApache /opt/oracle/9.2.0/Apache

cp -rp /opt/oracle/9.2.0/xApache /opt/oracle/9.2.0/Apache

Another problem. /home in the zone is a mount point, so you can't write directly

So we create yet another mount point

In the zone.

mkdir /realhomes

In /etc/vfstab:

/realhomes      -       /home                   lofs    -       yes     -

Seeing the global zone name from a zone

One quick method we use very successfully is to create a readonly lofs to /etc/nodename.
We add the following to all our zonecfgs

add fs
set dir=/etc/GLOBAL
set special=/etc/nodename
set type=lofs
add options [ro, nodevices]
end

so when your in a ngz you can cat /etc/GLOBAL to get the global host name.

Creating ZFS filesystems

The devices are hardware mirrored, so there isn't a huge advantage here, but Balatocan has 2 zfs filesystems created as:

zpool create -m /home homePool c3t0d0s7
zpool create -m /data dataPool c3t2d0s0

zfs create dataPool/openbis
zfs set mountpoint=/export/openbis dataPool/openbis
zfs set quota=4000G dataPool/openbis
zfs set sharenfs=on dataPool/openbis
zfs set sharenfs=rw=openbis.ethz.ch dataPool/openbis

Gromit has:

zpool create -m /home homePool c5t0d0s7

These persist across reboots.

Network configuration files

/etc/hostname.<interfaceName> - defines hostname/IP address/netmask/up or down
/etc/hosts - defines hostname
/etc/inet/netmasks - netmasks
/etc/inet/ntp.conf - ntp

Link aggregation

See: http://docs.sun.com/app/docs/doc/816-4554/gafxi?l=en&a=view&q=link+aggregation
The interfaces need to be unplumbed:

/etc # ifconfig e1000g3 unplumb 
/etc # ifconfig e1000g2 unplumb

Create the aggregation:
/etc # dladm create-aggr -d e1000g2 -d e1000g3 2
To make persistant, edit /etc/hostname.<interface>:
#/etc/hostname.aggr2
129.132.230.135
netmask 255.255.255.128
up

Jumbo frames

Changed sometime between snv_111 and snv_118.
You no longer enable jumbo frames in /kernel/drv/<driver>.conf, (except for the ixgb driver) you use dladm

dladm set-linkprop -p mtu=9194 <interface>

ixgb driver

In /kernel/drv/ixgb.conf add:

default_mtu = 9000;

Pre snv_118
To configure jumbo frames, you need to edit /kernel/drv/e1000g.conf and change MaxFramesize to 3 MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;.
Caution: This file may get overwritten by a live upgrade.
In /etc/hostname.aggr? add mtu 16128 and reboot.
Within the ETHZ network, leave the mtu at the default of 1500

On an x4540 with a 10Gbit ethernet card it's:
/kernel/drv/nxge.conf - add:

name = "pciex108e,abcd" parent = "/pci@79,0/pci10de,377@a" unit-address = "0"
accept_jumbo = 0;
name = "pciex108e,abcd" parent = "/pci@79,0/pci10de,377@a" unit-address = "0,1"
accept_jumbo = 1;

No need to put anything in /etc/hostname.nxge[01], they default to 1500 and 9194 respectively

Network configuration for the zones

The boxes that have 2 network interfaces, are configured with the main machine on the CISD server net (129.132.182.208/28) having a valid IP address so we can maintain it. The other interface is given a non-valid IP, but the interface must be up. Therefore, the interface files look like:

#/etc/hostname.aggr1
cisd-devsrv02
129.132.182.214
netmask 255.255.255.240
broadcast 129.132.182.223
up

#/etc/hostname.aggr2
0.0.0.0
up

Routing

Routing in a zone, where the zone does not have sufficient privileges to add a route, needs special consideration.
If a zone is bound to the aggr2 interface, with an IP address of say 129.132.230.137, the global zone does not know how to route for this network, and therefore nor does the zone.
To get around this problem, the zone is set to not autoboot, then a little start script on the global zone /etc/init.d/zone_route is called to:

  1. add 129.132.230.137 to aggr2
  2. add a default route via 129.132.230.129
  3. remove 129.132.230.137 from aggr2
  4. boot the zone
  5. In /etc/rc3.d a symlink (S99<zonename>) is created /etc/init.d/zone_route (The legacy way of starting a service)

Firewall Administration

Solaris uses ipfilter by Darren Reed. See http://coombs.anu.edu.au/ipfilter/
At CISD, firewall configuration files are under RCS control. see man rcs
Rules are in /etc/ipf/ipf.conf
Making changes:
Check out the file:

/etc/ipf # co -l ipf.conf

Edit /etc/ipf/ipf.conf with you favorite editor.
Check the file back in. Example:

/etc/ipf # ci -u -wjohn ipf.conf
RCS/ipf.conf,v  <--  ipf.conf
new revision: 1.9; previous revision: 1.8
enter log message, terminated with single '.' or end of file:
>> Added some firewall rule that blocks ferrets
>> .
done

At CISD, there is a script called ./reload.sh in /etc/ipf on each machine with firewall rules. It does the following for you:

  • # at now "+ 2min" < flush.sh otherwise you may get stuck without a network connection. (flush.sh just does ipf -Fa)
  • Restarts the rules with ipf -Fa -f ipf.conf (where -Fa deletes all the old rules before loading the new ones).
  • Lists all current rules with ipfstat -io and ipnat -l

    To list all currently set firewall and NAT rules, do:

    # ipfstat -io
    # ipnat -l
    

Firewall for NFS. There is a workaround here http://www.lildude.co.uk/2007/09/howto-dynamically-set-ipfilter-rules-for-rpc-services/

Stopping/starting a service

Services are controlled with the command svcadm. To find out what a service name is use svcs.
Here's an example of restarting sendmail:

biosa# svcs -a|grep mail
online         13:42:51 svc:/network/smtp:sendmail
biosa# svcadm restart svc:/network/smtp

Finding out what service (PID) binds to a port

The most portable way to do it is

# lsof -i

Works on Linux, Solaris and the BSDs.

pkginfo gives a list of installed packages. Add the -l <pkg> for more infomation
eg:

~ # pkginfo
...
application SUNWzfsgr                        ZFS Administration for Sun Java(TM) Web Console (Root)
application SUNWzfsgu                        ZFS Administration for Sun Java(TM) Web Console (Usr)
system      SUNWzfskr                        ZFS Kernel (Root)
...
~ # pkginfo -l SUNWzfsgu                                          
   PKGINST:  SUNWzfsgu
      NAME:  ZFS Administration for Sun Java(TM) Web Console (Usr)
  CATEGORY:  application
      ARCH:  i386
   VERSION:  1.0,REV=2007.03.12.08.57
   BASEDIR:  /
    VENDOR:  Sun Microsystems, Inc.
      DESC:  This package contains the user (/usr) component of the ZFS administration application for the Sun Java(TM) Web Console
    PSTAMP:  vx86-lx50e-blr0320070312085929
  INSTDATE:  Jun 27 2007 18:49
   HOTLINE:  Please contact your local service provider
    STATUS:  completely installed
     FILES:      310 installed pathnames
                   4 shared pathnames
                  31 directories
                3063 blocks used (approx)

installing a new package, eg from sunfreeware.com. Please put downloaded packages in /usr/local/src

/usr/local/src # pkgadd -d  openssl-0.9.8e-sol10-x86-local

The following packages are available:
  1  SMCossl     openssl
                 (x86) 0.9.8e

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [CISDIT:?,??,q]: 

"default" - just hit return
Removing a package

/usr/local/src # pkgrm SUNWzfsgu
## Verifying package <SUNWzfsgu> dependencies in zone <biosa>

The following package is currently installed:
   SUNWzfsgu  ZFS Administration for Sun Java(TM) Web Console (Usr)
              (i386) 1.0,REV=2007.03.12.08.57

Do you want to remove this package? [CISDIT:y,n,?,q] n

Packaging systems for Open Source software

There are a couple of alternatives there:

  • http://www.sunfreeware.com, which is the oldest one but some sometimes old versions and seems not to be very good when it comes to timely fixes for security updates
  • http://www.blastwave.org, which is newer site for the OpenSolaris community, looks quite up to date
  • http://www.netbsd.org/docs/software/packages.html, which is the NetBSD packaging system that is also available on Solaris, here everything is compiled from the source; this may require a long time and during this certain packages that use the same libraries are not operationsl
  • http://www.openpkg.org, uses RPM as its package format (i.e. uses binary packages); less packages that pkgsrc, but they claim that they have all the relevant stuff for servers; the claim to take security upgrades serious

Performing a Live Upgrade

Sun release a new version of Sun Developer Express edition every 3 months or so. To upgrade without having to take the machine down, you can use Live Update.
You need another partition the same size as the existing root partition. Here, I've used the swap partition.

  1. Download the latest "Solaris Express Developer Edition (sxde)" DVD
  2. Use a swapfile temporarily, by creating a swap file with:
     # mkfile 1g /var/swapfile
     # swap -a /var/swapfile
     # swap -l
     #dev/dsk/c5t0d0s1
    

    Format the old swap partition with 'format' and change the flag to wm from wu

  1. Remove the swap entry from /etc/vfstab
  2. You'll need to reboot or find another way to remount /tmp and /var/run
  3. I gave the new partitions the name sxde907 as in the first link
I hope this link still works for you: http://developers.sun.com/sxde/upgrade_guide.jsp
This one is a more complete/complex explanation http://docs.sun.com/app/docs/doc/819-6396/esqbt?a=view titled "Upgrading With Solaris Live Upgrade"

Multi path administration

A Solaris box with 2 FC HBAs will require failover. In the following we configure LUN 68 for failover with mpathadm.

  1. Look at the output from format: only relevant lines shown
bash-3.00# format
Searching for disks...

c3t5006048452A50667d68: configured with capacity of 505.75GB
c4t5006048452A50668d68: configured with capacity of 505.75GB

AVAILABLE DISK SELECTIONS:

 1. c3t5006048452A50667d68 <EMC-SYMMETRIX-5771 cyl 552418 alt 2 hd 15 sec 128>
/pci@0,0/pci1022,7458@11/pci10df,fc00@1/fp@0,0/disk@w5006048452a50667,44

 4. c4t5006048452A50668d68 <EMC-SYMMETRIX-5771 cyl 552418 alt 2 hd 15 sec 128>}}
/pci@0,0/pci1022,7458@11/pci10df,fc00@2/fp@0,0/disk@w5006048452a50668,44}}

Specify disk (enter its number): *1*
Disk not labeled.  Label it now? *n*
format> *format*
Ready to format.  Formatting cannot be interrupted
and takes 7673 minutes (estimated). Continue? *yes*
Beginning format. The current time is Thu Sep 27 08:55:37 2007

Formatting...
done
Verifying media...
pass 0 - pattern = 0xc6dec6de
7526/2/8

The EMC Symmetrix needs to be added to /kernel/drv/scsi_vhci.conf
Do a format -> enquiry and you'll get something like:

format> inquiry
Vendor:   EMC     
Product:  SYMMETRIX       
Revision: 5771

Now edit /kernel/drv/scsi_vhci.conf and add:

device-type-scsi-options-list =
"EMC     SYMMETRIX", "symmetric-option";
symmetric-option = 0x1000000;

Reboot.

Now format looks completely different - you'll only see half the number of disks. This is good, they are multipathed

~ # format          
Searching for disks...
Mode sense page(3) reports nsect value as 128, adjusting it to 128
done

c6t60060480000290100249533033343032d0: configured with capacity of 505.75GB
c6t60060480000290100249533033343345d0: configured with capacity of 1011.51GB
AVAILABLE DISK SELECTIONS:
       0. c5t0d0 <DEFAULT cyl 8872 alt 2 hd 255 sec 63>
          /pci@0,0/pci1022,7458@11/pci1000,3060@4/sd@0,0
       1. c6t60060480000290100249533033343032d0 <EMC-SYMMETRIX-5771 cyl 552418 alt 2 hd 15 sec 128>
          /scsi_vhci/disk@g60060480000290100249533033343032
       2. c6t60060480000290100249533033343230d0 <EMC-SYMMETRIX-5771 cyl 1679 alt 2 hd 255 sec 126>
          /scsi_vhci/disk@g60060480000290100249533033343230
       3. c6t60060480000290100249533033343345d0 <EMC-SYMMETRIX-5771 cyl 1104838 alt 2 hd 15 sec 128>
          /scsi_vhci/disk@g60060480000290100249533033343345
Specify disk (enter its number):

The disk that is formatted for Solaris is disk 2.

Next:

bash-3.00# luxadm display /dev/rdsk/c3t5006048452A50667d68s2 (s2 is whole disk)

Now run:

~ # mpathadm list lu
        /dev/rdsk/c6t60060480000290100249533033343345d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c6t60060480000290100249533033343230d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c6t60060480000290100249533033343032d0s2
                Total Path Count: 2
                Operational Path Count: 2

We want /dev/rdsk/c6t60060480000290100249533033343230d0s2 in this case.

Now run:

~ # mpathadm show lu /dev/rdsk/c6t60060480000290100249533033343230d0s2
Logical Unit:  /dev/rdsk/c6t60060480000290100249533033343230d0s2
        mpath-support:  libmpscsi_vhci.so
        Vendor:  EMC     
        Product:  SYMMETRIX       
        Revision:  5771
        Name Type:  unknown type
        Name:  60060480000290100249533033343230
        Asymmetric:  no
        Current Load Balance:  round-robin
        Logical Unit Group ID:  NA
        Auto Failback:  on
        Auto Probing:  NA

        Paths:  
                Initiator Port Name:  10000000c95d3b14
                Target Port Name:  5006048452a50667
                Override Path:  NA
                Path State:  OK
                Disabled:  no

                Initiator Port Name:  10000000c95d34f0
                Target Port Name:  5006048452a50668
                Override Path:  NA
                Path State:  OK
                Disabled:  no

        Target Ports:
                Name:  5006048452a50667
                Relative ID:  0

                Name:  5006048452a50668
                Relative ID:  0

Now continue following: http://docs.sun.com/source/819-0139/ch_3_admin_multi_devices.html

Set Font Path

If you get the following or similar error (I had it on my fresh installed VMWare)

Warning: Cannot convert string "-monotype-arial-regular-r-normal--*-140-*-*-p-*-iso8859-1" to type FontStruct
Warning: Unable to load any usable ISO8859-1 font
Warning:
    Name: FONTLIST_DEFAULT_TAG_STRING
    Class: XmRendition
    Conversion failed.  Cannot load font.

You have to set the font path. I found the solution in the VMWare Forum: http://communities.vmware.com/thread/104457

xset fp+ /usr/openwin/lib/X11/fonts/TTbitmaps

Enable SSL for apache2

SSL is disabled by default.
The following is an example of how to enable it for the Blastwave apache2, but the same is true of the built-in apache2

-bash-3.00# svcprop -p httpd/ssl svc:/network/http:cswapache2
false
-bash-3.00# svccfg -s svc:/network/http:cswapache2 setprop httpd/ssl=true
-bash-3.00# svcadm refresh svc:/network/http:cswapache2
-bash-3.00# svcprop -p httpd/ssl svc:/network/http:cswapache2
true

X11 forwarding not working on your Solaris 10 box?

This is because sshd is trying to use the ipv6 localhost (::1) by default which we don't have enabled.
The error message given in /var/adm/messages is:
Aug 23 11:25:09 cisd-egmont sshd[2685]: [ID 800047 auth.error] error: Failed to allocate internet-domain X11 display socket.

A quick fix is to run:
ifconfig lo0 inet6 plumb up
and to ensure the change persists across reboots:
touch /etc/hostname6.lo0

see http://spod.cx/blog/
or http://forum.java.sun.com/thread.jspa?threadID=5302236
On x86 Solaris, systems with patch 126134-04 exhibit this symptom - systems with 126134-02 do not.

How to set cron's default PATH

Edit /etc/default/cron to look like:

CRONLOG=YES
PATH=/usr/sbin:/usr/bin:/opt/csw/sbin:/opt/csw/bin
SUPATH=/usr/sbin:/usr/bin:/opt/csw/sbin:/opt/csw/bin

Restart and refresh cron

~# svcadm disable svc:/system/cron:default
~# svcadm refresh svc:/system/cron:default
~# svcadm enable svc:/system/cron:default

The SUPATH is for root's crontab

Sharing a zfs filesystem

Open port 2049/tcp for the clients that need access
On server:

root@cisd-krakatoa # zfs set mountpoint=/export dataPool
root@cisd-krakatoa # zfs create dataPool/imsb-openbis
root@cisd-krakatoa # zfs set quota=4000G dataPool/imsb-openbis
root@cisd-krakatoa # zfs set sharenfs=on dataPool/imsb-openbis
root@cisd-krakatoa # zfs set sharenfs=rw=imsb-openbis.ethz.ch dataPool/imsb-openbis
root@cisd-krakatoa # zfs get sharenfs

Another example:
root@bs-ssvr01 # zfs set sharenfs='rw=@129.132.27.0/26,rw=@129.132.228.0/26,root=bs-dsvr01-s.ethz.ch' dataPool/bsse/bsse-it

On client:

root@cisd-ruapehu # mount -o proto=tcp,public cisd-krakatoa:/export/imsb-openbis /mnt/krakatoa

or

# echo "/misc           auto_misc       -nosuid,nobrowse" >> /etc/auto_master
# echo "ruapehu -fstype=nfs,rw,proto=tcp        thumper2:/export/imsb-openbis" >> /etc/auto_misc
# svcadm disable autofs
# svcadm enable autofs

Here's the performance:

/mnt/krakatoa/data $ time dd if=/dev/zero of=1G bs=1024k count=1000
1000+0 records in
1000+0 records out

real    0m11.008s
user    0m0.005s
sys     0m2.626s
(95Mbytes/sec)
/mnt/krakatoa/data $ time dd if=/dev/zero of=10G bs=1024k count=10000
10000+0 records in
10000+0 records out

real    1m36.347s
user    0m0.053s
sys     0m27.586s
(108Mbytes/sec)

Rsync setup using rsync from Blastwave.org

On the master thumper, add the following to /etc/inetd.conf

rsync   stream  tcp     nowait  root    /opt/csw/bin/rsync      rsyncd --daemon

and the following to /etc/services

rsync           873/tcp                         # Rsync daemon

Then run

~# inetconv 

to put it under control of the SMF

If you get this error:

# inetconv
inetconv: Error /etc/inet/inetd.conf line 44 invalid or inconsistent fields: service-name protocol

Check you have the correct entry in /etc/services

Set up /etc/rsyncd.conf to look like:

log file = /var/log/rsyncd.log

[imsb-openbis]
path=/export/imsb-openbis
comment = Source of files from master thumper
hosts allow = thumper2
read only = yes

Enable rsyncd using the SMF

# svcadm enable svc:/network/inetd
# svcadm enable svc:/network/rsync/tcp

Check SMF entry

# svcprop -p inetconv/source_line rsync/tcp
rsync\ stream\ tcp\ nowait\ root\ /opt/csw/bin/rsync\ rsyncd\ --daemon

From the slave thumper, do

#!/bin/bash
RSYNC=/opt/csw/bin/rsync

$RSYNC -av --delete --rsync-path=$RSYNC thumper1::imsb-openbis  /export/imsb-openbis/

Snapshot and save a filesystem

# export DATE=`/bin/date +%Y-%m-%d_%H%M`
# zfs snapshot dataPool/imsb-openbis@$DATE
# zfs list  -t snapshot
NAME                                    USED  AVAIL  REFER  MOUNTPOINT
dataPool/imsb-openbis@2008-01-23_1055      0      -  13.3G  -
# zfs send dataPool/imsb-openbis@2008-01-23_1055 > /misc/ruapehu/snapshot.$DATE

Then on remote host (ruapehu)

# cat snapshot.2008-01-23_1055 | zfs recv dataPool/test3@today # creates a new zfs filesystem at /dataPool/test3

How to find out the memory model of a Solaris system

Call

~# isainfo -b

The output will be 64 or 32, depending on the memory model of the system.

Measure network throughput

# kstat -p 'link:0:aggr1:rbytes64' 1

Add a persistent route.

Example.
If you have 2 interfaces on the same network, but want to route packets to a particular host via a particular interface.

       Host A                        Host B
 _____________________         ____________________
| nxge1 172.31.45.165 | ----> | nge1 172.31.45.167 |
|                     |       |____________________|             Host C
|                     |                                      _____________________     
| aggr0 172.31.45.171 | --------------------------------- > | nxge1 172.31.45.164 |
|_____________________|                                     |_____________________|
                                                            
~ # route -p add 172.31.45.167 172.31.45.165  -interface
add persistent host 172.31.45.167: gateway 172.31.45.165
~ # route -p add 172.31.45.164 172.31.45.171  -interface
add persistent host 172.31.45.164: gateway 172.31.45.171

View persistent routes

cat /etc/inet/static_routes
# File generated by route(1M) - do not edit.
172.31.53.64 10.130.87.225 255.255.255.192
192.168.123.32 10.130.87.225 255.255.255.224
192.168.128.64 10.130.87.225 255.255.255.192
192.168.123.16 10.130.87.225 255.255.255.240
172.31.45.167 172.31.45.165 -interface
172.31.45.164 172.31.45.171 -interface
192.168.197.80 10.130.87.225 255.255.255.240