$Cambridge: hermes/doc/cyrus_mailstore/Backup/otanes,v 1.2 2003/11/03 17:08:05 dpc22 Exp $ Otanes configuration Guide ========================== Concept ======= 16 live Cyrus systems replicate their mailstore to the Hermes backup spooling engine (named Otanes: brother-in-law of Cyrus) at regular intervals: at least daily, possibly several times a day. Otanes converts filesystems into linear dump images and streams them to tape. Hardware Overview ================= Dual CPU Intel system currently running RedHat 9 - SR2300 chassis, SUV2 motherboard with onboard SCSI - 2 x 2.4 GByte Pentium IV Xeon (Hyperthreading currently enabled in BIOS) Qlogic QLA2312 PCI to Fibre Channel Host Bus Adaptor - Single channel - BIOS enabled so that we can boot off device "scsi(0:0:0:0)" Infortrend EonStor A16F Fibre to SATA RAID box - 16 x 250 GByte Maxtor disk - configured as 15 disk RAID 5 set + hot spare - split into two paritions (=> LUNs) to work around 2 TByte block device limit in Linux 2.4 kernel (will no longer be needed with 2.6). Overland Powerloader tape stacker - 2 x LTO-2 (aka Ultrium-2) tape drive - 17 tape bays. Will be split up as: 2 x 8 tape dump cycle: managed by mtx utility: see below slot for cleaning tape: can be reserved, used automatically 3 Ethernet Interfaces (all Intel EtherPro 1000, two on motherboard + card) eth0: 172.28.13.60 -- Cambridge Wide Private Network for Cyrus systems. eth1: 192.168.128.124 -- Hermes Intramail network eth2: 192.168.0.60 -- Loopback connection to Disk box (*) (*) Replace Crossover with -intramail link Software Overview ================= Redhat 9 + Logical Volume Manager (LVM) + SGI XFS patch XFS has better large file performance than ReiserFS, worse small file performance (but nothing approaches ReiserFS for small file performance). We use XFS because of the xfsdump and xfsrestore commands. MTX used to control Tape stacker: http://mtx.badtux.net/ Installation and initial postinstall ==================================== System is installed using Kickstart floppy: - ucvs/hermes/kickstart/otanes_dir/9/ks.cfg Run Postinstall scripts from /misc/kickstart/cyrus_dir/9/scripts: (NFS automount mount of prism-intramail:/export/kickstart/...) - setup-users.sh :: Creates user accounts and groups - setup-programs.sh :: Configures system programs - patch.sh :: Downloads RPM updates from ftp-uxsup.csx.cam.ac.uk --> /var/tmp/postinstall Install custom Redhat Kernel from oss.sgi.com to get XFS support. ftp://oss.sgi.com/.pub0/xfs/Release-1.3.1/cmd_rpms ftp://oss.sgi.com/.pub0/xfs/Release-1.3.1/kernel_rpms cached at: /misc/kickstart/cyrus_dir/9/xfs/ updates :: of already installed packages install :: xfsdump and friends kernel :: the main event Make sure that correct kernel will boot from /boot/grub/grub.conf /etc/modules.conf needs a "max_scsi_luns=8" alias eth0 e1000 alias eth1 e1000 alias scsi_hostadapter qla2300 alias scsi_hostadapter1 aic79xx alias usb-controller usb-uhci options scsi_mod.o max_scsi_luns=8 Remake the initial ram disk: will include max_scsi_luns from modules.conf: # cd /boot # mkinitrd initrd-2.4.20-20.9smp.img 2.4.20-20.9 Reboot the system: Updated kernel will pick up both LUNs as sda and sdb Nov 3 14:07:37 otanes kernel: Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 Nov 3 14:07:37 otanes kernel: Attached scsi disk sdb at scsi0, channel 0, id 0, lun 1 Disk partitioning ================= Use fdisk to create physical volumes for LVM: Device Boot Start End Blocks Id System /dev/sda1 * 1 522 4192933+ 83 Linux /dev/sda2 523 783 2096482+ 83 Linux /dev/sda3 784 1044 2096482+ 82 Linux swap /dev/sda4 1045 210345 1681210282+ 8e Linux LVM Device Boot Start End Blocks Id System /dev/sdb4 1 216409 1738305261 8e Linux LVM Notes: - Linux LVM is ID 8e. - May need to reboot again before kernel will pick up new partition table for boot device: sdb seems to work fine. LVM setup ========= Create a pair of physical volumes on /dev/sda4 and /dev/sdb4: # pvcreate /dev/sda4 /dev/sdb4 Create a volume group which combines the two physical volumes. # vgcreate -s 32m otanes /dev/sda4 /dev/sdb4 - Physical extent of "32m" required if we want to create logical volumes up to 2 TBytes in size. Default physical extent of "4m" only allows us to create logical volumes up to 256 GBytes in size. Not a problem given that all the partitions that we create are many GBytes in size (and in fact "32m" reduces LVM overhead considerably by reduce the number of indirection blocks). Create a couple of virtual tape buffers first, so that they are at the outside edge of the disks for maximum performancs (135 Mbyte/sec at the outside verus 105 MBytes/sec for logical volumes at the inside of the disk) # lvcreate -n tape0 -L 375g otanes # lvcreate -n tape1 -L 375g otanes Create 16 logical volumes as targets for Cyrus replication: # for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 > do > lvcreate -n $i -L 144g otanes > done This leaves space for an additional 144g scratch partition if we ever need to reorganise the disk, plus 63 Gbytes for snapshots: otanes[root]# vgdisplay --- Volume group --- VG Name otanes VG Access read/write VG Status available/resizable VG # 0 MAX LV 256 Cur LV 19 Open LV 18 MAX LV Size 2 TB Max PV 256 Cur PV 2 Act PV 2 VG Size 3.18 TB PE Size 32 MB Total PE 104352 Alloc PE / Size 102336 / 3.12 TB Free PE / Size 2016 / 63 GB VG UUID 37L58d-RB7L-f0ZH-6w6v-sutn-R1Ie-sTbWQ3 Filesystem setup ================ # for i in tape0 tape1 1 2 3 4 5 6 7 8 9 10 11 12 12 13 14 15 16 > do > mkfs -t xfs /dev/otanes/$i > done Which gives us to following to play with: otanes[dpc22]$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 4127076 981116 2936316 26% / none 1549204 0 1549204 0% /dev/shm /dev/sda2 2063536 75888 1882824 4% /var /dev/otanes/tape0 393084928 44845524 348239404 12% /spool/tape/0 /dev/otanes/tape1 393084928 1520 393083408 1% /spool/tape/1 /dev/otanes/1 150921216 6196 150915020 1% /spool/cyrus/1 /dev/otanes/2 150921216 39624280 111296936 27% /spool/cyrus/2 /dev/otanes/3 150921216 2177020 148744196 2% /spool/cyrus/3 /dev/otanes/4 150921216 2048684 148872532 2% /spool/cyrus/4 /dev/otanes/5 150921216 2389932 148531284 2% /spool/cyrus/5 /dev/otanes/6 150921216 2210268 148710948 2% /spool/cyrus/6 /dev/otanes/7 150921216 1492392 149428824 1% /spool/cyrus/7 /dev/otanes/8 150921216 1636264 149284952 2% /spool/cyrus/8 /dev/otanes/9 150921216 1176 150920040 1% /spool/cyrus/9 /dev/otanes/10 150921216 1176 150920040 1% /spool/cyrus/10 /dev/otanes/11 150921216 1176 150920040 1% /spool/cyrus/11 /dev/otanes/12 150921216 1176 150920040 1% /spool/cyrus/12 /dev/otanes/13 150921216 1176 150920040 1% /spool/cyrus/13 /dev/otanes/14 150921216 1176 150920040 1% /spool/cyrus/14 /dev/otanes/15 150921216 1176 150920040 1% /spool/cyrus/15 /dev/otanes/16 150921216 1176 150920040 1% /spool/cyrus/16 Also see: ./fstab Cyrus Setup =========== XXX Account for /etc/cyrus/imapd-hermes-*.conf # mount -a # # chown dump:dump /spool/tape0 /spool/tape1 # chmod 2750 /spool/tape0 /spool/tape1 # # for i in 1 2 3 4 5 6 7 8 9 10 11 12 12 13 14 15 16 > do > chown cyrus:cyrus /spool/cyrus/$i > chgrp 750 /spool/cyrus/$i > done As Cyrus user: $ for i in 1 2 3 4 5 6 7 8 9 10 11 12 12 13 14 15 16 > do > # Running in subshell! > cd /spool/cyrus/$i > mkdir dbase log mailstore sieve > chmod 750 dbase log mailstore sieve > /usr/local/bin/mkimap /etc/cyrus/imapd-$i.conf > done Replication =========== XXX Add wrapper script here As Cyrus user (cyrus users on otanes, cyrus-X use SSH RSA authenication): $ for i in 1 2 3 4 5 6 7 8 > do > ssh cyrus-$i replicate -s otanes -R /etc/cyrus/imapd-$i.conf \ -u -v `cat /var/map-hermes/accounts/master` > done Creating dump images ==================== See ./dump.sh XXX Improve Streaming dump images to tape ============================= See ./tar.sh. Key point is block size of 1000 blocks (512 KBytes) in order to get the LTOs streaming at full speed without shoe shine effects. XXX Improve XXX XXX Possible to use dd rather than tar, so that we can xfsrestore XXX straight from tape?