######################################################################## # # Presentation about migrating from CUS to Hermes # Wednesday 2004-01-21 # # $Cambridge: hermes/doc/talks/2004-01-techlinks/talk.mgp,v 1.17 2004/01/21 13:58:10 fanf2 Exp $ # ######################################################################## # %deffont "standard" tfont "standard.ttf", size 5 %deffont "thick" tfont "thick.ttf" %deffont "typewriter" tfont "typewriter.ttf" # %default 1 area 90 90, leftfill, size 2, fore "black", back "white", font "thick" %default 2 size 7, vgap 10, prefix " " %default 3 size 2, bar "gray70", vgap 10 %default 4 fore "black", vgap 30, prefix " ", font "standard", size 5 # %tab 1 prefix " ", icon box "gray30" 50 # ######################################################################## %page %nodefault, center, fore "black", back "white", size 5 %newimage -zoom 100 "../../misc/orange-fire/dimms.jpg" # # One of the reasons we want to get rid of Old Hermes... # ######################################################################## %page %nodefault, center, fore "black", back "white", font "thick", size 8 Migrating email from CUS to Hermes %size 6 TechLinks 2004-01-21 %font "standard", size 5 Tony Finch Mail Support University of Cambridge Computing Service # # General plan of talk: # Reasons for migrating # Migration run-through # Migrating from other systems # ######################################################################## %page POP and IMAP %area 40 50 10 25 %center %newimage -zoom 100 "pop.jpg" %area 40 50 50 30 %center %newimage -zoom 100 "imac.jpg" # # Migrating from CUS is only necessary for POP and IMAP users. # People who read their email on CUS using Pine or MH etc. # are not affected and do not have to move to Hermes. # People who use Pine on CUS to read their Hermes email are not affected. # # Sorry about the dreadful pun. # ######################################################################## %page Withdrawal timetable No POP for new users since May 1999 No IMAP for new users since Oct. 2003 Every so often POP & IMAP are disabled for people who haven't used them recently Complete withdrawal at end of August 2004 # # This is not a new policy. # # Much easier to finish the process now that Hermes is so much better. # ######################################################################## %page Problems of POP and IMAP on CUS UNIX format email folders perform badly Slow to parse a folder into messages Slow to remove older messages Exacerbated by frequent polling for new email and large mailbox quotas # # CUS has greater compatibility restrictions than Hermes # so it can't use a better folder format # # Old Hermes uses the same folder format, but augments it with indexes # to speed up opening mailboxes, and restricts mailbox size to keep # deleting messages fast. # # New Hermes is much more clever. # ######################################################################## %page Advantages of new Hermes %center %newimage -zoom 200 "storage.jpg" %size 7 100+MB quotas # # Lots of storage -- about 40 times that of old Hermes # so we can be even more generous with quotas than CUS. # ######################################################################## %page Advantages of new Hermes %area 40 40 10 20 %center %newimage -zoom 100 "storage.jpg" %area 40 40 50 20 %center %newimage -zoom 100 "storage.jpg" %area 40 40 50 55 %center %newimage -zoom 100 "storage.jpg" %area 40 40 10 55 %center %newimage -zoom 100 "storage.jpg" # # Data replication # ######################################################################## %page Advantages of new Hermes Very efficient mailbox format heavily indexed for fast opens full-text index for fast searches Data is kept for a month messages and folders can be undeleted replacement for NetApp snapshots even works for POP Data is replicated in 4 places hot replica on paired machine warm copy on disk backup cold copy on tape in safe # # As well as large total quotas, large mailboxes are not a problem. # The 10MB mailbox limit on old Hermes confused people. # # CUS and old Hermes use NetApp snapshots to make it possible to # recover from accidental deletion, though coverage is not complete # -- messages deleted between snapshots are not backed up, which # is a particular problem for POP users. # # If we have annother "thermal event" it shouldn't disrupt service too # much. In the future we hope to have a second machine room so Hermes # can keep working even if the fire alarm is set off. # ######################################################################## %page CUS vs. Hermes - questions? %center %newimage -zoom 100 "query-1.png" ######################################################################## %page Migration %center %newimage -zoom 100 "wildebeest.jpg" # # We have to move over 30,000 accounts from old to new Hermes. # About half way there now. # 1000 CUS users still to be moved. # ######################################################################## %page Before migration Move Hermes account from old to new system a mostly transparent operation users will be notified of planned move accounts will be moved later in batches confirmation message sent after move then you can run the migration tool New accounts are still created on old Hermes they should be moved before migration contact user-admin@ucs if necessary www.cam.ac.uk/cs/email/hermes_migration.html # # We'll send out the notification message to CUS users this week. # When an account is migrated users receive a confirmation message, # after which they can move their email from CUS to Hermes. # user-admin@ucs.cam.ac.uk is the contact for old -> new # http://www.cam.ac.uk/cs/email/hermes_migration.html # ######################################################################## %page Migration process Route email to Hermes ensure Hermes doesn't redirect email away arrange for @cam email to go to Hermes arrange for @cus email to go to Hermes Upload saved email from CUS to Hermes must be done over IMAP mailbox names may need to be altered # # Most of the later steps can be done via Hermes webmail and Jackdaw, # but it's fiddly. # # Redirection fixing has to be done in the right order # and it's easy to forget to fix one of the three places. # # New Hermes doesn't store mailboxes in a conventional file system, # so ftp and scp can't be used. # # Mailbox names may need to be renamed because they clash or because # they need to be trimmed. # # A special tool can collect together all the operations # into an easy package. # ######################################################################## %page A tool: 2hermes %center %newimage -zoom 200 "leatherman-flair.jpg" # ######################################################################## %page Intro and authentication %center %newimage -zoom 200 "intro-auth.png" # # I've tried to make it friendly, # though you should not run it unless you mean to! # # The warning is not a last chance -- # it's quite easy to reverse the forwarding changes # ######################################################################## %page Routing changes - simple case %center %newimage -zoom 200 "forward-simple.png" # # This is the simple case where all email is delivered to CUS # and there is no filtering on CUS. # Note that it describes what it has done and how to undo it. # Undoing the changes on Hermes isn't handled. # ######################################################################## %page Routing changes - common case %center %newimage -zoom 200 "forward-spam.png" # # More common is the same as above, but with a spam filter on CUS. # Note the extra text about fiddling the local .forward file # and how to set up filtering on Hermes. # ######################################################################## %page Routing changes - afterwards %center %newimage -zoom 200 "forward-spam-after.png" # # After 2hermes has run the .forward files look like this. # Same thing happens if you have more complicated .forward files. # ######################################################################## %page Routing changes - complicated %center %newimage -zoom 200 "forward-advanced.png" # # This is an example of what 2hermes does when it finds an advanced # sieve script and a weird @cam forwarding destination. # # After the advanced sieve script has been disabled, # it is still available via the webmail interface. # # It also confirms @cam changes if the dest was neither CUS nor Hermes. # ######################################################################## %page Routing changes - notes Hermes has simple interface to filters suitable for most users Advanced users can still write their own Exim filters on CUS Sieve filters on Hermes http://www.cyrusoft.com/sieve/ RFC 3028 The software tries hard not to lose data keeps backup of old filters Can safely be run more than once ######################################################################## %page Routing changes - questions? %center %newimage -zoom 200 "query-2.jpg" ######################################################################## %page Uploading saved email %center %newimage -zoom 150 "filing-cabinet.jpg" # # The other thing 2hermes does is copy people's saved email to Hermes. # ######################################################################## %page Uploading saved email Search for mailboxes in usual places /var/mail/fanf9 ~/inbox ~/mbox ~/mail/ ~/Mail/ Canonicalize folder names /var/mail/fanf9 -> INBOX ~/mail/saved-messages -> saved-messages Avoid folder name clashes folder with same name exists on server upload folder with different name or append messages to folder # # If the program finds the wrong files, move the ones you want it # to upload into one of the right places, and move the ones you # don't want away. # # The canonicalization is to fix names according to IMAP terminology. # # The user is given the choice of appending messages to any folders # that already exist on Hermes, or to rename local folders so they # don't clash. This is done by appending a number. # ######################################################################## %page Uploading email - clash handling %center %newimage -zoom 200 "upload-start.png" # # So the first question you are asked is about folder renaming. # # I'll repeat the same example with both yes and no answers. # ######################################################################## %page Uploading email - webmail (before) %center %newimage -zoom 100 "webmail-before.png" # # Both examples start with an empty Hermes account like this. # ######################################################################## %page Uploading email - renamed folders %center %newimage -zoom 200 "upload-clash-y.png" # # First it lists the folders on the server. # .prayer is the webmail preferences file. # The curly brackets are an artefact of the IMAP library c-client. # # The second list is what it's proposing to do. # ######################################################################## %page Uploading email - renamed folders %center %newimage -zoom 200 "upload-done-y.png" # # After you confirm it goes ahead, finishing with a friendly message. # ######################################################################## %page Uploading email - webmail (after) %center %newimage -zoom 100 "webmail-after-y.png" # # And the result looks like this in webmail. # # At this point you can rename the folders # if you don't like the names chosen by 2hermes. # ######################################################################## %page Uploading email - merged folders %center %newimage -zoom 200 "upload-clash-n.png" # # So if you answer no to the question it doesn't try to avoid clashes. # # Note that there are two local folders called onemsg which will # be merged. If you don't want this to happen, press CTRL+C and # rename the folders before running 2hermes again. # ######################################################################## %page Uploading email - merged folders %center %newimage -zoom 200 "upload-done-n.png" ######################################################################## %page Uploading email - webmail (after) %center %newimage -zoom 100 "webmail-after-n-folders.png" ######################################################################## %page Uploading email - webmail (after) %center %newimage -zoom 100 "webmail-after-n-onemsg.png" # # This folder contains messages from Mail/onemsg and mail/onemsg # ######################################################################## %page Uploading lots of saved email %center %newimage -zoom 100 "movable-stacks.jpg" # # Some people have quite a lot of saved email. # Or they haven't moved to new Hermes yet! # ######################################################################## %page Uploading email - quota problems %center %newimage -zoom 200 "upload-quota.png" # # 2hermes warns if you have lots of saved email on CUS. # This warning occurs at 80MB of email if your account is on new Hermes. # # If you run 2hermes before you have been moved to the new system, # the trigger point is at 8MB which is much easier to reach. # It's best to wait until you get the confirmation message. # # If you have problems with uploading, you can always re-run 2hermes. # ######################################################################## %page Uploading email - subdirectories %center %newimage -zoom 200 "upload-subdir.png" # # This example is also interesting because # it illustrates what 2hermes does with subdirectories. # ######################################################################## %page Uploading email - webmail (after) %center %newimage -zoom 100 "webmail-subdir-1.png" ######################################################################## %page Uploading email - webmail (after) %center %newimage -zoom 100 "webmail-subdir-2.png" ######################################################################## %page Uploading email - questions? %center %newimage -zoom 100 "query-3.jpg" ######################################################################## %page Inside 2hermes %center %newimage -zoom 150 "guts.png" # # For those interested in installing 2hermes on systems other than CUS. # ######################################################################## %page Inside 2hermes %center %newimage -zoom 210 "2hermes.eps" # # Left hand half deals with email routing changes. # Right hand half is for uploading saved email. # # Most of the forwarding changes are done by a daemon # which runs on the Hermes admin box. # This is necessary because of security restrictions in access to # Jackdaw's cammail service and the Hermes Sieve management interface # and for access to old Hermes .forward files. # The daemon chats in a friendly manner, albeit over SSL (to protect passwords), # so it could easily be used by software other than 2hermes. # The vertical bars in the forwarding fix-up part of the 2hermes # output indicate which part is run remotely on Hermes. # # Most of the work of uploading mailboxes is done by the imapupload # program. 2hermes locates the mailboxes that should be uploaded, and # imapupload does any renaming that is necessary and then performs # the upload. It uses c-client and OpenSSL to do the hard work. # There are more options to imapupload than 2hermes exposes. # # Source code can be found on CUS in /opt/2hermes/src # and you can email mail-support@ucs.cam.ac.uk with questions. # ######################################################################## %page 2hermes in general - questions? %center %newimage -zoom 100 "query-4.jpg" ######################################################################## %page And now for something completely different %center %newimage -zoom 200 "foot.jpg" # # That's all about migration. # # Before I finish a little bit about our ongoing work. # ######################################################################## %page Mail support work in progress New message store commission second tranche of servers re-write user admin scripts purchase new login servers eventually decommission old Hermes Upgrade ppsw email hub support for roaming Hermes users firewall-friendly IP addresses better spam filtering # # Some delays because of power supply problems. # Users are still being added to old Hermes # Before we can decommission old Hermes we need new computers # to run the menu system and Pine etc. # # We'll support SMTP AUTH for relaying control, so you'll be able # to use the same email software setup anywhere. # ppsw moved into a dedicated address range for simpler firewall setups. # Upgrade SpamAssassin and MailScanner. # ######################################################################## %page That's all, folks %center %newimage -zoom 100 "hermes.jpg" http://www.cus.cam.ac.uk/~fanf2/ mail-support@ucs.cam.ac.uk # ########################################################################