Solaris tips and tricks - wiki
- Attachments:1
- Added by Ryan John, last edited by Ryan John on Dec 05, 2010 (view change)
- show comment
-
Solaris 10
- Creating a new Zone
- Creating ZFS filesystems
- Network configuration files
- Link aggregation
- Jumbo frames
- Network configuration for the zones
- Routing
- Firewall Administration
- Stopping/starting a service
- Finding out what service (PID) binds to a port
- Packaging systems for Open Source software
- Performing a Live Upgrade
- Multi path administration
- Set Font Path
- Enable SSL for apache2
- X11 forwarding not working on your Solaris 10 box?
- How to set cron's default PATH
- Sharing a zfs filesystem
- Rsync setup using rsync from Blastwave.org
- Snapshot and save a filesystem
- How to find out the memory model of a Solaris system
- Measure network throughput
- Add a persistent route.
- View persistent routes
Solaris 10
Creating a new Zone
bash-3.00# zonecfg -z zone1 imsb-lims: No such zone configured Use 'create' to begin configuring a new zone. zonecfg:zone1> create zonecfg:zone1> set zonepath=/home/zones/zone1 zonecfg:zone1> set autoboot=false zonecfg:zone1> add net zonecfg:zone1:net> zonecfg:zone1:net> zonecfg:zone1:net> zonecfg:zone1:net> set address=129.132.230.223 zonecfg:zone1:net> set physical=aggr2 zonecfg:zone1:net> end zonecfg:zone1> add fs zonecfg:zone1:fs> set dir=/data zonecfg:zone1:fs> set special=/data/openbis zonecfg:zone1:fs> set type=lofs zonecfg:zone1:fs> add options [nodevices] zonecfg:zone1:fs> end zonecfg:zone1> exit bash-3.00# zoneadm -z zone1 verify bash-3.00# zoneadm -z zone1 install bash-3.00# zoneadm -z zone1 boot bash-3.00# zlogin -e \; -C zone1 # The -C option to zlogin specifies the target zone. The -e option to zlogin specifies the escape character. Ie: to escape - do ;.
After login, complete the normal Solaris install. (Answer the questions)
To create a read/write sudirectory in a read-only filesystem:
Example: In order that the zones can write to /opt/oracle/9.2.0/dbs
In zone
mkdir /oracledbs mount -F lofs /oracledbs /opt/oracle/9.2.0/dbs
Put the following entry in /etc/vfstab:
/oracledbs - /opt/oracle/9.2.0/dbs lofs - yes -
In order that a zone can write to an existing directory.
Example: being able to change config files in /opt/oracle/9.2.0/Apache
In global zone
mv /opt/oracle/9.2.0/Apache /opt/oracle/9.2.0/xApache mkdir /opt/oracle/9.2.0/Apache # make a mountpoint only
In zone
mkdir /oraApache mount -F lofs /oraApache /opt/oracle/9.2.0/Apache cp -rp /opt/oracle/9.2.0/xApache /opt/oracle/9.2.0/Apache
Another problem. /home in the zone is a mount point, so you can't write directly
So we create yet another mount point
In the zone.
mkdir /realhomes In /etc/vfstab: /realhomes - /home lofs - yes -
Seeing the global zone name from a zone
One quick method we use very successfully is to create a readonly lofs to /etc/nodename.
We add the following to all our zonecfgs
add fs set dir=/etc/GLOBAL set special=/etc/nodename set type=lofs add options [ro, nodevices] end
so when your in a ngz you can cat /etc/GLOBAL to get the global host name.
Creating ZFS filesystems
The devices are hardware mirrored, so there isn't a huge advantage here, but Balatocan has 2 zfs filesystems created as:
zpool create -m /home homePool c3t0d0s7 zpool create -m /data dataPool c3t2d0s0 zfs create dataPool/openbis zfs set mountpoint=/export/openbis dataPool/openbis zfs set quota=4000G dataPool/openbis zfs set sharenfs=on dataPool/openbis zfs set sharenfs=rw=openbis.ethz.ch dataPool/openbis
Gromit has:
zpool create -m /home homePool c5t0d0s7
These persist across reboots.
Network configuration files
/etc/hostname.<interfaceName>
- defines hostname/IP address/netmask/up or down/etc/hosts
- defines hostname/etc/inet/netmasks
- netmasks/etc/inet/ntp.conf
- ntp
Link aggregation
See: http://docs.sun.com/app/docs/doc/816-4554/gafxi?l=en&a=view&q=link+aggregation
The interfaces need to be unplumbed:
/etc # ifconfig e1000g3 unplumb /etc # ifconfig e1000g2 unplumb
Create the aggregation:/etc # dladm create-aggr -d e1000g2 -d e1000g3 2
To make persistant, edit /etc/hostname.<interface>:#/etc/hostname.aggr2
129.132.230.135
netmask 255.255.255.128
up
Jumbo frames
Changed sometime between snv_111 and snv_118.
You no longer enable jumbo frames in /kernel/drv/<driver>.conf
, (except for the ixgb driver) you use dladm
dladm set-linkprop -p mtu=9194 <interface>
ixgb driver
In /kernel/drv/ixgb.conf
add:
default_mtu = 9000;
Pre snv_118
To configure jumbo frames, you need to edit /kernel/drv/e1000g.conf and change MaxFramesize to 3 MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;.
Caution: This file may get overwritten by a live upgrade.
In /etc/hostname.aggr? add mtu 16128 and reboot.
Within the ETHZ network, leave the mtu at the default of 1500
On an x4540 with a 10Gbit ethernet card it's:/kernel/drv/nxge.conf
- add:
name = "pciex108e,abcd" parent = "/pci@79,0/pci10de,377@a" unit-address = "0" accept_jumbo = 0; name = "pciex108e,abcd" parent = "/pci@79,0/pci10de,377@a" unit-address = "0,1" accept_jumbo = 1;
No need to put anything in /etc/hostname.nxge[01]
, they default to 1500 and 9194 respectively
Network configuration for the zones
The boxes that have 2 network interfaces, are configured with the main machine on the CISD server net (129.132.182.208/28) having a valid IP address so we can maintain it. The other interface is given a non-valid IP, but the interface must be up. Therefore, the interface files look like:
#/etc/hostname.aggr1
cisd-devsrv02
129.132.182.214
netmask 255.255.255.240
broadcast 129.132.182.223
up
#/etc/hostname.aggr2
0.0.0.0
up
Routing
Routing in a zone, where the zone does not have sufficient privileges to add a route, needs special consideration.
If a zone is bound to the aggr2 interface, with an IP address of say 129.132.230.137, the global zone does not know how to route for this network, and therefore nor does the zone.
To get around this problem, the zone is set to not autoboot, then a little start script on the global zone /etc/init.d/zone_route is called to:
- add 129.132.230.137 to aggr2
- add a default route via 129.132.230.129
- remove 129.132.230.137 from aggr2
- boot the zone
- In /etc/rc3.d a symlink (S99<zonename>) is created /etc/init.d/zone_route (The legacy way of starting a service)
Firewall Administration
Solaris uses ipfilter by Darren Reed. See http://coombs.anu.edu.au/ipfilter/
At CISD, firewall configuration files are under RCS control. see man rcs
Rules are in /etc/ipf/ipf.conf
Making changes:
Check out the file:
/etc/ipf # co -l ipf.conf
Edit /etc/ipf/ipf.conf
with you favorite editor.
Check the file back in. Example:
/etc/ipf # ci -u -wjohn ipf.conf RCS/ipf.conf,v <-- ipf.conf new revision: 1.9; previous revision: 1.8 enter log message, terminated with single '.' or end of file: >> Added some firewall rule that blocks ferrets >> . done
At CISD, there is a script called ./reload.sh
in /etc/ipf
on each machine with firewall rules. It does the following for you:
-
# at now "+ 2min" < flush.sh
otherwise you may get stuck without a network connection. (flush.sh just doesipf -Fa
) - Restarts the rules with
ipf -Fa -f ipf.conf
(where-Fa
deletes all the old rules before loading the new ones). - Lists all current rules with
ipfstat -io
andipnat -l
To list all currently set firewall and NAT rules, do:
# ipfstat -io # ipnat -l
Firewall for NFS. There is a workaround here http://www.lildude.co.uk/2007/09/howto-dynamically-set-ipfilter-rules-for-rpc-services/
Stopping/starting a service
Services are controlled with the command svcadm. To find out what a service name is use svcs.
Here's an example of restarting sendmail:
biosa# svcs -a|grep mail online 13:42:51 svc:/network/smtp:sendmail biosa# svcadm restart svc:/network/smtp
Finding out what service (PID) binds to a port
The most portable way to do it is
# lsof -i
Works on Linux, Solaris and the BSDs.
pkginfo gives a list of installed packages. Add the -l <pkg> for more infomation
eg:
~ # pkginfo ... application SUNWzfsgr ZFS Administration for Sun Java(TM) Web Console (Root) application SUNWzfsgu ZFS Administration for Sun Java(TM) Web Console (Usr) system SUNWzfskr ZFS Kernel (Root) ... ~ # pkginfo -l SUNWzfsgu PKGINST: SUNWzfsgu NAME: ZFS Administration for Sun Java(TM) Web Console (Usr) CATEGORY: application ARCH: i386 VERSION: 1.0,REV=2007.03.12.08.57 BASEDIR: / VENDOR: Sun Microsystems, Inc. DESC: This package contains the user (/usr) component of the ZFS administration application for the Sun Java(TM) Web Console PSTAMP: vx86-lx50e-blr0320070312085929 INSTDATE: Jun 27 2007 18:49 HOTLINE: Please contact your local service provider STATUS: completely installed FILES: 310 installed pathnames 4 shared pathnames 31 directories 3063 blocks used (approx)
installing a new package, eg from sunfreeware.com. Please put downloaded packages in /usr/local/src
/usr/local/src # pkgadd -d openssl-0.9.8e-sol10-x86-local The following packages are available: 1 SMCossl openssl (x86) 0.9.8e Select package(s) you wish to process (or 'all' to process all packages). (default: all) [CISDIT:?,??,q]:
"default" - just hit return
Removing a package
/usr/local/src # pkgrm SUNWzfsgu ## Verifying package <SUNWzfsgu> dependencies in zone <biosa> The following package is currently installed: SUNWzfsgu ZFS Administration for Sun Java(TM) Web Console (Usr) (i386) 1.0,REV=2007.03.12.08.57 Do you want to remove this package? [CISDIT:y,n,?,q] n
Packaging systems for Open Source software
There are a couple of alternatives there:
- http://www.sunfreeware.com, which is the oldest one but some sometimes old versions and seems not to be very good when it comes to timely fixes for security updates
- http://www.blastwave.org, which is newer site for the OpenSolaris community, looks quite up to date
- http://www.netbsd.org/docs/software/packages.html, which is the NetBSD packaging system that is also available on Solaris, here everything is compiled from the source; this may require a long time and during this certain packages that use the same libraries are not operationsl
- http://www.openpkg.org, uses RPM as its package format (i.e. uses binary packages); less packages that pkgsrc, but they claim that they have all the relevant stuff for servers; the claim to take security upgrades serious
Performing a Live Upgrade
Sun release a new version of Sun Developer Express edition every 3 months or so. To upgrade without having to take the machine down, you can use Live Update.
You need another partition the same size as the existing root partition. Here, I've used the swap partition.
- Download the latest "Solaris Express Developer Edition (sxde)" DVD
- Use a swapfile temporarily, by creating a swap file with:
# mkfile 1g /var/swapfile # swap -a /var/swapfile # swap -l #dev/dsk/c5t0d0s1
Format the old swap partition with
'format'
and change the flag towm
fromwu
- Remove the swap entry from /etc/vfstab
- You'll need to reboot or find another way to remount /tmp and /var/run
- I gave the new partitions the name sxde907 as in the first link
This one is a more complete/complex explanation http://docs.sun.com/app/docs/doc/819-6396/esqbt?a=view titled "Upgrading With Solaris Live Upgrade"
Multi path administration
A Solaris box with 2 FC HBAs will require failover. In the following we configure LUN 68 for failover with mpathadm.
- Look at the output from format: only relevant lines shown
bash-3.00# format Searching for disks... c3t5006048452A50667d68: configured with capacity of 505.75GB c4t5006048452A50668d68: configured with capacity of 505.75GB AVAILABLE DISK SELECTIONS: 1. c3t5006048452A50667d68 <EMC-SYMMETRIX-5771 cyl 552418 alt 2 hd 15 sec 128> /pci@0,0/pci1022,7458@11/pci10df,fc00@1/fp@0,0/disk@w5006048452a50667,44 4. c4t5006048452A50668d68 <EMC-SYMMETRIX-5771 cyl 552418 alt 2 hd 15 sec 128>}} /pci@0,0/pci1022,7458@11/pci10df,fc00@2/fp@0,0/disk@w5006048452a50668,44}} Specify disk (enter its number): *1* Disk not labeled. Label it now? *n* format> *format* Ready to format. Formatting cannot be interrupted and takes 7673 minutes (estimated). Continue? *yes* Beginning format. The current time is Thu Sep 27 08:55:37 2007 Formatting... done Verifying media... pass 0 - pattern = 0xc6dec6de 7526/2/8
The EMC Symmetrix needs to be added to /kernel/drv/scsi_vhci.conf
Do a format -> enquiry and you'll get something like:
format> inquiry Vendor: EMC Product: SYMMETRIX Revision: 5771
Now edit /kernel/drv/scsi_vhci.conf
and add:
device-type-scsi-options-list = "EMC SYMMETRIX", "symmetric-option"; symmetric-option = 0x1000000;
Reboot.
Now format
looks completely different - you'll only see half the number of disks. This is good, they are multipathed
~ # format Searching for disks... Mode sense page(3) reports nsect value as 128, adjusting it to 128 done c6t60060480000290100249533033343032d0: configured with capacity of 505.75GB c6t60060480000290100249533033343345d0: configured with capacity of 1011.51GB AVAILABLE DISK SELECTIONS: 0. c5t0d0 <DEFAULT cyl 8872 alt 2 hd 255 sec 63> /pci@0,0/pci1022,7458@11/pci1000,3060@4/sd@0,0 1. c6t60060480000290100249533033343032d0 <EMC-SYMMETRIX-5771 cyl 552418 alt 2 hd 15 sec 128> /scsi_vhci/disk@g60060480000290100249533033343032 2. c6t60060480000290100249533033343230d0 <EMC-SYMMETRIX-5771 cyl 1679 alt 2 hd 255 sec 126> /scsi_vhci/disk@g60060480000290100249533033343230 3. c6t60060480000290100249533033343345d0 <EMC-SYMMETRIX-5771 cyl 1104838 alt 2 hd 15 sec 128> /scsi_vhci/disk@g60060480000290100249533033343345 Specify disk (enter its number):
The disk that is formatted for Solaris is disk 2.
Next:
bash-3.00# luxadm display /dev/rdsk/c3t5006048452A50667d68s2 (s2 is whole disk)
Now run:
~ # mpathadm list lu /dev/rdsk/c6t60060480000290100249533033343345d0s2 Total Path Count: 2 Operational Path Count: 2 /dev/rdsk/c6t60060480000290100249533033343230d0s2 Total Path Count: 2 Operational Path Count: 2 /dev/rdsk/c6t60060480000290100249533033343032d0s2 Total Path Count: 2 Operational Path Count: 2
We want /dev/rdsk/c6t60060480000290100249533033343230d0s2
in this case.
Now run:
~ # mpathadm show lu /dev/rdsk/c6t60060480000290100249533033343230d0s2 Logical Unit: /dev/rdsk/c6t60060480000290100249533033343230d0s2 mpath-support: libmpscsi_vhci.so Vendor: EMC Product: SYMMETRIX Revision: 5771 Name Type: unknown type Name: 60060480000290100249533033343230 Asymmetric: no Current Load Balance: round-robin Logical Unit Group ID: NA Auto Failback: on Auto Probing: NA Paths: Initiator Port Name: 10000000c95d3b14 Target Port Name: 5006048452a50667 Override Path: NA Path State: OK Disabled: no Initiator Port Name: 10000000c95d34f0 Target Port Name: 5006048452a50668 Override Path: NA Path State: OK Disabled: no Target Ports: Name: 5006048452a50667 Relative ID: 0 Name: 5006048452a50668 Relative ID: 0
Now continue following: http://docs.sun.com/source/819-0139/ch_3_admin_multi_devices.html
Set Font Path
If you get the following or similar error (I had it on my fresh installed VMWare)
Warning: Cannot convert string "-monotype-arial-regular-r-normal--*-140-*-*-p-*-iso8859-1" to type FontStruct Warning: Unable to load any usable ISO8859-1 font Warning: Name: FONTLIST_DEFAULT_TAG_STRING Class: XmRendition Conversion failed. Cannot load font.
You have to set the font path. I found the solution in the VMWare Forum: http://communities.vmware.com/thread/104457
xset fp+ /usr/openwin/lib/X11/fonts/TTbitmaps
Enable SSL for apache2
SSL is disabled by default.
The following is an example of how to enable it for the Blastwave apache2, but the same is true of the built-in apache2
-bash-3.00# svcprop -p httpd/ssl svc:/network/http:cswapache2 false -bash-3.00# svccfg -s svc:/network/http:cswapache2 setprop httpd/ssl=true -bash-3.00# svcadm refresh svc:/network/http:cswapache2 -bash-3.00# svcprop -p httpd/ssl svc:/network/http:cswapache2 true
X11 forwarding not working on your Solaris 10 box?
This is because sshd is trying to use the ipv6 localhost (::1) by default which we don't have enabled.
The error message given in /var/adm/messages is:Aug 23 11:25:09 cisd-egmont sshd[2685]: [ID 800047 auth.error] error: Failed to allocate internet-domain X11 display socket.
A quick fix is to run:ifconfig lo0 inet6 plumb up
and to ensure the change persists across reboots:touch /etc/hostname6.lo0
see http://spod.cx/blog/
or http://forum.java.sun.com/thread.jspa?threadID=5302236
On x86 Solaris, systems with patch 126134-04 exhibit this symptom - systems with 126134-02 do not.
How to set cron's default PATH
Edit /etc/default/cron to look like:
CRONLOG=YES PATH=/usr/sbin:/usr/bin:/opt/csw/sbin:/opt/csw/bin SUPATH=/usr/sbin:/usr/bin:/opt/csw/sbin:/opt/csw/bin
Restart and refresh cron
~# svcadm disable svc:/system/cron:default ~# svcadm refresh svc:/system/cron:default ~# svcadm enable svc:/system/cron:default
The SUPATH is for root's crontab
Sharing a zfs filesystem
Open port 2049/tcp for the clients that need access
On server:
root@cisd-krakatoa # zfs set mountpoint=/export dataPool root@cisd-krakatoa # zfs create dataPool/imsb-openbis root@cisd-krakatoa # zfs set quota=4000G dataPool/imsb-openbis root@cisd-krakatoa # zfs set sharenfs=on dataPool/imsb-openbis root@cisd-krakatoa # zfs set sharenfs=rw=imsb-openbis.ethz.ch dataPool/imsb-openbis root@cisd-krakatoa # zfs get sharenfs Another example: root@bs-ssvr01 # zfs set sharenfs='rw=@129.132.27.0/26,rw=@129.132.228.0/26,root=bs-dsvr01-s.ethz.ch' dataPool/bsse/bsse-it
On client:
root@cisd-ruapehu # mount -o proto=tcp,public cisd-krakatoa:/export/imsb-openbis /mnt/krakatoa
or
# echo "/misc auto_misc -nosuid,nobrowse" >> /etc/auto_master # echo "ruapehu -fstype=nfs,rw,proto=tcp thumper2:/export/imsb-openbis" >> /etc/auto_misc # svcadm disable autofs # svcadm enable autofs
Here's the performance:
/mnt/krakatoa/data $ time dd if=/dev/zero of=1G bs=1024k count=1000 1000+0 records in 1000+0 records out real 0m11.008s user 0m0.005s sys 0m2.626s (95Mbytes/sec) /mnt/krakatoa/data $ time dd if=/dev/zero of=10G bs=1024k count=10000 10000+0 records in 10000+0 records out real 1m36.347s user 0m0.053s sys 0m27.586s (108Mbytes/sec)
Rsync setup using rsync from Blastwave.org
On the master thumper, add the following to /etc/inetd.conf
rsync stream tcp nowait root /opt/csw/bin/rsync rsyncd --daemon
and the following to /etc/services
rsync 873/tcp # Rsync daemon
Then run
~# inetconv
to put it under control of the SMF
If you get this error:
# inetconv inetconv: Error /etc/inet/inetd.conf line 44 invalid or inconsistent fields: service-name protocol
Check you have the correct entry in /etc/services
Set up /etc/rsyncd.conf to look like:
log file = /var/log/rsyncd.log [imsb-openbis] path=/export/imsb-openbis comment = Source of files from master thumper hosts allow = thumper2 read only = yes
Enable rsyncd using the SMF
# svcadm enable svc:/network/inetd # svcadm enable svc:/network/rsync/tcp
Check SMF entry
# svcprop -p inetconv/source_line rsync/tcp rsync\ stream\ tcp\ nowait\ root\ /opt/csw/bin/rsync\ rsyncd\ --daemon
From the slave thumper, do
#!/bin/bash RSYNC=/opt/csw/bin/rsync $RSYNC -av --delete --rsync-path=$RSYNC thumper1::imsb-openbis /export/imsb-openbis/
Snapshot and save a filesystem
# export DATE=`/bin/date +%Y-%m-%d_%H%M` # zfs snapshot dataPool/imsb-openbis@$DATE # zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT dataPool/imsb-openbis@2008-01-23_1055 0 - 13.3G - # zfs send dataPool/imsb-openbis@2008-01-23_1055 > /misc/ruapehu/snapshot.$DATE
Then on remote host (ruapehu)
# cat snapshot.2008-01-23_1055 | zfs recv dataPool/test3@today # creates a new zfs filesystem at /dataPool/test3
How to find out the memory model of a Solaris system
Call
~# isainfo -b
The output will be 64
or 32
, depending on the memory model of the system.
Measure network throughput
# kstat -p 'link:0:aggr1:rbytes64' 1
Add a persistent route.
Example.
If you have 2 interfaces on the same network, but want to route packets to a particular host via a particular interface.
Host A Host B _____________________ ____________________ | nxge1 172.31.45.165 | ----> | nge1 172.31.45.167 | | | |____________________| Host C | | _____________________ | aggr0 172.31.45.171 | --------------------------------- > | nxge1 172.31.45.164 | |_____________________| |_____________________|
~ # route -p add 172.31.45.167 172.31.45.165 -interface add persistent host 172.31.45.167: gateway 172.31.45.165 ~ # route -p add 172.31.45.164 172.31.45.171 -interface add persistent host 172.31.45.164: gateway 172.31.45.171
View persistent routes
cat /etc/inet/static_routes # File generated by route(1M) - do not edit. 172.31.53.64 10.130.87.225 255.255.255.192 192.168.123.32 10.130.87.225 255.255.255.224 192.168.128.64 10.130.87.225 255.255.255.192 192.168.123.16 10.130.87.225 255.255.255.240 172.31.45.167 172.31.45.165 -interface 172.31.45.164 172.31.45.171 -interface 192.168.197.80 10.130.87.225 255.255.255.240