A few months ago we transitioned from hosting all of our apps in development on two Slicehost slices to individual Amazon EC2 instances for each. As much you want vendor frozen Rails and Merb to work (including all your app's dependencies) it's always plagued with issues. Having completely separate environment for each app clears up a lot of issues, including the all to often mistake of upgrading Rails only to find some old project isn't able to co-exist with it. Our EC2 setup involved creating a base image with the basic Rails environment, and whenever a new staging server was needed we would clone the Base image and modify it specifically for the app. Being able to startup new environments with a single command is wonderfully easy, but a dozen staging servers later, the cost certainly starts to add up.

So for the cost of 1 month of our EC2 instances, we purchased a Dell box with 8Gb to run as an internal staging server and host all of our virtual servers. We chose Ubuntu as both the host and guest operating system. We started out trying KVM which is officially supported by Ubuntu. Everything went to plan and we had a great setup except for one major problem: slices would randomly lock up. After no success finding a remedy we decided to give Xen a go and with a little work got it setup and running along nicely.

Most of the instructions here have come from a combination of the Ubuntu and Virtuatopia wikis.

The first step was to install Ubuntu 8.10 (Intrepid) Server Edition (remember to select the "Server Edition" tab before downloading).

Setup Xen

Detailed instructions are on the Ubuntu wiki Xen page

sudo apt-get install ubuntu-xen-server

Install Xen kernel from Hardy

Grab the .deb from here: http://packages.ubuntu.com/hardy-updates/linux-image-2.6.24-21-xen

sudo dpkg -i linux-image-2.6.24-21-xen_2.6.24-21.43_amd64.deb

Make sure your eth0 is configured.

Reboot.

Using xen-tools to create images

We should now have an Ubuntu server running the Xen kernel with all the standard Xen tools installed.

Next we'll create a base Xen image running Debian Etch. In an ideal world it would be running Ubuntu 8.10 (Intrepid), but there isn't yet a xen-tools template for Ubuntu, so Debian has to do.

Configure xen-tools:

vim /etc/xen-tools/xen-tools.conf

This is our config (installing Debian Etch):

dir = ~/slices
install-method = debootstrap

size   = 4Gb      # Disk image size.
memory = 256Mb    # Memory size
swap   = 512Mb    # Swap size
fs     = ext3     # use the EXT3 filesystem for the disk image.
dist   = etch     # Default distribution to install.
image  = sparse   # Specify sparse vs. full disk images.

dhcp = 1

passwd = 1

accounts = 1

kernel      = /boot/vmlinuz-`uname -r`
initrd      = /boot/initrd.img-`uname -r`

arch = amd64

mirror = http://ftp.us.debian.org/debian/

ext3_options   = noatime,nodiratime,errors=remount-ro
ext2_options   = noatime,nodiratime,errors=remount-ro
xfs_options    = defaults
reiser_options = defaults

If you want to include anything extra in the image, you can put files in the /etc/xen-tools/skel directory. We've written a script to configure the slice for Rails with Apache running Phusion Passenger.

Grab the boot_xen_passenger.sh script, and copy it to /etc/xen-tools/skel if you plan on using it.

sudo vim /etc/xen-tools/skel/boot_xen_passenger.sh 
sudo chmod +x /etc/xen-tools/skel/boot_xen_passenger.sh

Create your first slice:

mkdir ~/slices
sudo xen-create-image --hostname=xentest1

Start image:

sudo xm create /etc/xen/xentest1.cfg

View slices with:

sudo xm list

Attach to the console with:

sudo xm console xentest1

SSH in and run the setup script:

sudo ./boot_xen_passenger.sh

Networking

Hostname resloving

Dnsmasq is a nice little dns server which will allow us to reference our slices by their hostnames.

sudo apt-get install dnsmasq

Dnsmasq can also act as a dhcp server. When you slices ask for a dhcp lease they will also give dnsmasq their hostname. This way you won't need to reserve ip addresses for slices. However we've had some issues with the dhcp client on Debian when used in conjunction dnsmasq. In some cases you will want to forward certain ports from the outside world to a slice, and if have a dumb router it will only be able to forward to a specific ip.

If you wish you setup the dhcp server in dnsmasq:

sudo vim /etc/dnsmasq.conf

Uncommend the line #dhcp-range=192.168.0.50,192.168.0.150,12h and edit the ip range if desired.

If you use a separate router for your internet, you'll want to set the gateway options as well by uncommenting the following line dhcp-option=option:router,192.168.1.1 and replacing 1.2.3.4 with your router's ip.

Now restart dnsmasq:

sudo /etc/init.d/dnsmasq restart

If you have an existing dhcp server on the network you will need to change it to forward requests to the new dnsmasq dhcp server. This is to ensure that new slices ask dnsmasq for an ip address, and when doing so it will also broadcast its hostname. On the local network this will allow you do to nice things like http://myslice and ssh myslice.

However if you've opted not use dnsmasq as your dhcp server, you will need to add a line to the host's /etc/hosts file with the hostname of the slice and its ip address (every time you edit /etc/hosts you will also need to restart dnsmasq). You will also need to specify the ip address of your slices when they are created. See the Configuring Xen Guest Network Options wiki page for more information about configuring xen-tools.

On your router or for individual clients you will also need to add the ip of the server with dnsmasq setup as a DNS server.

Magic subdomain forwarding with nginx

Any slice we startup should now be accessible by its hostname. To reduce the configuration required for each slice, we setup Nginx on the host to automatically forward http requests from any subdomain under our staging domain to the slice with a hostname the same as the subdomain. So that if you visit http://myslice.mystagingdomain.com from anywhere, it will be internally redirected to http://myslice and allow all your slices to be behind one ip.

Setup nginx

Grab nginx:

sudo apt-get install nginx

Then add a new site config for the slice forwarding:

sudo vim /etc/nginx/sites-available/slices_proxy.conf

Here is the config we have:

map $host $backend {
    hostnames;
}

server {
  resolver 127.0.0.1;
  listen   80;
  server_name  _;

  access_log  /var/log/nginx/localhost.access.log;

  location / {
    if ($host ~* "^(.*).mystagingdomain.co.uk") {
      set $host_without_www $1;
      proxy_pass http://$host_without_www$request_uri;
    }
  }
}

Enable the config:

sudo ln -s /etc/nginx/sites-available/slices_proxy.conf /etc/nginx/sites-enabled/slices_proxy.conf

Best to also remove the default config:

sudo rm /etc/nginx/sites-enabled/default

Then restart nginx:

sudo /etc/init.d/nginx restart

Setup DNS

Next add an wildcard A record to your mystagingdomain.com domain to point to your host's external ip, and all should be good to go.

Next steps

The instructions here should get you started with a local Xen setup. However there's still a few of the finer points to perfect. The Rails setup script could be integrated with Xen better so it's automatically run when a slice is created. The script itself is also very brittle and likely to fail on another environment. Using deprec would probably be a better solution in the long term.