Migrating Websites Between Servers

Last updated: May 31, 2016 — Joel Dueck (@joeld)

0. Preamble

This is a boiled-down version of a sheet that I wrote for my own use over the course of a couple of server migrations. The migration instructions apply equally well to shared or VPS hosting, but they do assume you have shell access on both hosts. They also don’t assume access to web-based admin tools like virtualmin or phpmyadmin.

A note about email: I don’t mess with hosting my own email server. Trust me, you don’t want to either (if you do then you’re not the type of person who will find this guide useful). Offload your email needs onto Fastmail or Google Apps or Outlook 365 (a great option these days). You will be much happier. At the very least do not host your own email unless you switch to a VPS. People trying to host their own email on shared hosting are the #1 cause of server slowdowns and crashes, in my own experience.

1. Virtual Private Server vs. Shared Hosting

Shared hosting: It’s cheap. You’re going to have trouble with shared hosting, and it will often be trouble caused by other people that you can do nothing about. If you’re trying to run a business site, or if anyone besides you is dependent on the sites you run, a Virtual Private Server (VPS) is going to give you a vastly better experience. The only downside to a VPS is that you have to do much of the work setting up and configuring the server yourself (remotely, through shell access). I’ve linked to some great resources below for helping you wit this. Once you configure your web server, there is hardly any upkeep.

Managed VPS: If you want a don’t want to manage your server environment, Minneapolis-based Arcustech is my number one all-around recommendation. You get shell access and the reliability of your own dedicated CPU/RAM (no “shared hosting”), plus the ease of a completely managed environment by a company with a full support staff. They’ve been in business for fifteen years. Jeffrey Zeldman recommends them. You can get a 1 CPU box with 512MB of RAM for $12.50 USD/month with their Lite VPS package. If you go this route, ArcusTech will handle everything in section 2 below, and some operations (such as creating new mysql databases) require a support ticket, but otherwise everything in section 3 applies.

Plain VPS: With most VPS providers you are your own sysadmin. You get a bare-bones OS installation and get to set everything up from there. It’s cheap, reliable (except for your own mistakes) and you learn a lot. Most people use either Linode or DigitalOcean.

Personally I went with a $5 VPS from Digital Ocean. With this setup, running Textpattern with caching enabled on top of Apache, I’ve handled a couple of days where I was getting 250 pageviews an hour (such as after this happened) with no problem. I also I ran a simulation of 1,000 hits with a maximum 100 concurrent from another server (here’s one way to do this). Memory usage maxed out at 250MB, but server load climbed right up to 3.1 (which is a bit high for a single-core VPS). Hits were served anywhere from 1.3 to 2.1 seconds during this test. As of this writing my uptime is 103 days (I haven’t needed to restart the VPS since I first opened the account).

2. Setting up a VPS

This section is here in case you decide to go the VPS route as opposed to shared. It requires a lot more work when you first set things up, but it gives you more control over your environment, and insulates you from the effects of unruly neighbors.

2.1. Basic VPS Configuration

Here are my top three resources for setting up a VPS:

Read at least the first two links above. This is stuff you need to de to keep from getting hacked and to be sure you can keep an eye on things.

2.1.1. Possible Thing You Might Want: Automated Setup with `ansible`

Suppose this is your second or third time around setting up a new VPS. You’re pulling up the above posts again, trying to remember what packages to install and what configuration files to edit. There must be a faster way, you think.

Well there is. Ansible is a tool for automating server configuration. Some of the highlights:

Ansible uses playbooks, which are plain text files that list tasks in a very readable form. Tasks can do just about anything: install packages, ensure certain lines are added to or edited within any configuration file, set up users and passwords, create databases, etc.
Ansible playbooks thus serve a dual purpose as both scripts and as a human-readable documentation of configuration steps.
Ansible uses ssh to make changes, so no special agent or software needs to be installed on the server for it to work.

I’ve created an Ansible playbook that automates all the steps described in the first two links above: securing your server, configuring MySQL, Apache and PHP, and setting up configs and folders for individual websites. It even automatically installs free trusted SSL certificates from LetsEncrypt (something not yet described in this guide).

If you’ve set up a few servers and are tired of doing it by hand every time, it is highly worth your while to learn Ansible.

2.2 Tuning Your VPS

After you get your VPS set up, depending on how much traffic you expect, you may want to try your hand at tuning it for better perfomance — primarily if you are serving more than just static HTML files and have a smaller VPS (e.g. <1 GB of RAM). If that doesn’t describe you, just skip to section 2.3.

2.2.1. Tuning Apache

Note:

This section is only worth doing if you are using the prefork module as Apache’s multi-processing module (MPM). If not, skip to 2.2.2.
- If you are using Apache’s mod_php module to serve PHP files, then you must necessarily be using prefork as your MPM.
- See How do I select which Apache MPM to use? for a good explainer of the various MPMs.

Got it? Alrighty.

The default multi-processing module (MPM) for Apache 2.4 is the event module, so to use the prefork module, we need to edit /etc/apache2/mods-available/mpm_prefork.conf. But first we need to find some roughly optimal values to use.

Run top and press SHIFT+M. Note the highest RES memory used by apache2.
Hit Q to exit top, then run sudo service apache2 stop

Now run free -m and note the memory listed under “used”. In the example below, the value we want is 134 (i.e., ignore memory used for caching):

             total       used       free     shared    buffers     cached
Mem:           490        444         45          5         42        267
-/+ buffers/cache:        134        355
Swap:            0          0          0

Subtract the memory used from your VPS’s nominal memory (e.g. 512MB or 1GB). This will give you your base free memory pool.
Multiply the value of your free memory pool by 0.8 — this will give a good average available Apache pool (and allow you a 20% memory reserve for burst periods).
Divide your available Apache pool by the highest RES memory used by apache2 (step 1 above). Round to the nearest integer. We’ll call this “MaxClients”.

Now edit /etc/apache2/mods-available/mpm_prefork.conf and use the following:

Set MaxRequestWorkers to the MaxClients value calculated above
Set MinSpareServers to 10–25% of MaxClients
Set MaxSpareServers to roughly 25–50% of MaxClients
Set your StartServers equal to either MinSpareServers or MaxSpareServers. When apache is restarted, this is the number of servers that will start and be ready for connections immediately. High-traffic sites should set this value to MaxSpareServers and lower volume sites should use MinSpareServers.
Set MaxConnectionsPerChild higher if you don’t have memory leaks, lower if you see rapid memory growth with Apache processes.

Here is what I use on my 512MB VPS:

<IfModule mpm_prefork_module>
    StartServers            5
    MinSpareServers         5
    MaxSpareServers         10
    MaxRequestWorkers       22
    MaxConnectionsPerChild  3000
</IfModule>

Also edit /etc/apache2/apache2.conf:

Set KeepAlive to Off if you don’t need it (your server will handle more requests per second with keepalive on, but will require more memory for apache–don’t turn on KeepAlive if you’re not leaving a 20% memory reserve).
If you are leaving KeepAlive on, set KeepAliveTimeout to the lowest value you can to prevent connections from hanging. If you experience high latency to your server, set KeepAliveTimeout to 2–5 seconds.
Set your Timeout to a reasonable value. Pick a time that won’t cut off the transfer of your pages to your customers, but keep it low enough that you don’t have dead connections that remain open for a large period of time. 10 seconds is a reasonable value; the default of 300 is ridiculous.
Set MaxKeepAliveRequests to the largest number of objects you have in 1 page. If you don’t know, pick something in the range of 70–200.

After finishing these edits:

sudo a2dismod mpm_event
sudo a2enmod mpm_prefork

Finally remember to sudo service apache2 start (if still stopped from step 2 above, otherwise use restart at the end instead).

2.2.2. Tuning MySQL

For now I have not changed any of the default settings.

2.2.3. Tuning PHP

Once PHP5 is installed, tune the configuration file located in /etc/php5/apache2/php.ini to enable more descriptive errors, logging, and better performance. The following modifications provide a good starting point — find the relevant lines further down in the file and edit them as follows:

error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR  
error_log = /var/log/php/error.log  
max_input_time = 30

Then create the log directory for PHP and give the Apache user ownership:

sudo mkdir /var/log/php
sudo chown www-data /var/log/php
sudo service apache2 reload

2.3 Terminal customizations

I like to set up Oh My ZSH:

sudo apt-get install zsh
sudo apt-get install git
curl -L https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh | sh

nano ~/.zshrc     # switch theme to "agnoster"
nano ~/.oh-my-zsh # edit prompt_context to use custom colors, emoji

3. Migrating Your Sites

Once you can log in to your new host, and you have Apache, PHP and MySQL running, you can start migrating.

First, set each domain’s DNS timeouts to something short, like 60 seconds.

Second, it’s a good idea, though not critical, to create a non-root user account within MySQL:

mysql> CREATE USER 'newuser'@'localhost' IDENTIFIED BY 'password';

The instructions below assume you’ve taken this step. (More information on this)

Now, here’s what you do for each site.

3.1. Prepare

On the old server, export your database to a file (so it will be included with the file transfer in the next step):

 # Export MySQL database
mysqldump -u username -pPASSWORD db_name > dbbackup.sql

3.2. On the New Server

Prepare website folders. The examples here set up each domain in its own folder underneath a www folder in your own directory.

mkdir -p ~/www/site.com/web/public
mkdir /www/site.com/logs

Create your database user account in MySQL — open MySQL console with mysql -u root -p:

mysql> CREATE DATABASE db_name;
Query OK, 1 row affected (0.01 sec)

mysql> GRANT ALL PRIVILEGES ON db_name . * to 'dbuser'@'localhost';
Query OK, 0 rows affected (0.01 sec)

3.3. Make Your Move

Now import all your files. Rsync allows you to do so directly between the two servers. (Note the final period in the rsync command.)

cd ~/www/site.com/web/public
rsync -ave ssh username@oldserver.net:domains/site.com/web/public/ .

# Import your DB
mysql -u dbuser -pPASSWORD dbname < dbbackup.sql

# Edit your CMS config files for new DB name/credentials
# (Example below is for Textpattern)
# Make sure dbcharset is properly configured for Textpattern! (utf8)
nano ~/www/site.com/web/public/textpattern/config.php

3.4. Configure Site in Apache

Open the site’s virtual host file with sudo nano /etc/apache2/sites-available/site.com and include the following (updated to reflect Apache 2.4, see here for more info):

<VirtualHost *:80>  
     ServerAdmin you@site.com  
     ServerName site.com  
     ServerAlias *.site.com  
     DocumentRoot /home/user/www/site.com/web/public/  
     ErrorLog /home/user/www/site.com/logs/error.log  
     CustomLog /home/user/www/site.com/logs/access.log combined  
<Directory /home/user/www/site.com/web/public>  
     Options -Indexes +FollowSymLinks  
     AllowOverride All  
     Require all granted  
</Directory>  
</VirtualHost>

Then:

# Switch the website to 'enabled' status
sudo a2ensite site.com

# Reload the service
sudo service apache2 reload

3.5. Setting up logrotate

If not already installed, run sudo apt-get install logrotate.

You can add your own logs to be rotated by editing /etc/logrotate.conf. Here’s what I use for each of my sites:

/home/deployuser/www/domain.com/logs/*.log {
    missingok
    monthly
    rotate 12
    compress
    delaycompress
    notifempty
}

3.6. Migrating Piwik

I use Piwik for self-hosted web stats – just because I don’t want to give my users’ info to Google Analytics. Here’s a quick outline of how to migrate your piwik MySQL database.

First put piwik in “maintenance mode” (disable tracking) by adding this to config/config.ini.php:

[Tracker]  
record_statistics = 0