Aegir-up: a Vagrant project templating framework

!!! WARNING: This blog post contains code !!!

Quite a bit of it, actually. It's in Ruby. I've never coded in Ruby before this, so in all likelihood it sucks. But I probably won't be able to recognize that until I've had more experience hacking on Ruby code (maybe in Vagrant or Puppet). Also, my academic and professional background is in philosophy and marketing, so there's no help there. Anyway, consider yourself warned. That said, I'm very open to constructive criticism, so feel free to make suggestions.

So, what's this all about then?

I discovered Vagrant about 6 months ago, when a friend and colleague suggested I try it out, as a way to learn Puppet. I fell in love with both technologies. Just like when I discovered Drupal (~5 years ago), I was able to start using them productively almost immediately, but I also found an easy on-ramp to delving deeper.

I'd wanted to learn Puppet at the time in order to further automate deployment and management of Aegir servers, as we ramped up Koumbit's AegirVPS service. I also wanted to be able to test things like mass migration of sites between Aegir servers, and Vagrant provided a perfect tool to do both.

Once the Aegir Puppet module started to come together, it became clear that deploying an Aegir server locally, within Vagrant, was worthwhile in it's own right. And so, Aegir-up was born.

Initially, I found it challenging to find the relevant bits in a Vagrantfile to allow me to alter settings. So I began to abstract my Vagrantfile, and isolate all settings in an included 'settings.rb' file. This resulted in an .ini-like control file. Then I started getting frustrated as I'd forget which settings had to be tweaked with each new project, to avoid collisions and such. So, I started to learn Bash scripting in order to automate project initialization.

Automating project creation implied the use of a template, which then begged to be further generalized. So, I ended up with a template that would deploy an Aegir server from the Debian packages, and another that would install everything "manually" directly from the Git repositories. Along the way, I discovered Veewee, and started bundling customized base box definitions within Aegir-up.

At this point, things started to get a little out-of-hand. The scripts ended up with bits of perl, sed and other bits of Unix tools embedded within them, along with duplicated argument handling code, and such. Seeing as how this was essentially a Drupal-specific tool, I heeded the advice of one of Aegir's maintainers, Steven Jones, and re-wrote the scripts as a Drush extension. this was a great opportunity to further delve into the intricacies of Drush, since most of the heavy lifting in Aegir is done in a Drush extension, Provision.

Steven also pointed out that the most powerful piece of Aegir-up was this templating system, that could be generalized into a framework. Lately, I've been working on just that "frameworkifying", and it's almost there. I'm now considering adding a Rake script to handle some of the template work, and just calling that script from within Drush.This would have the benefit of making the template framework independent of Drush, and thus potentially useful for the broader Vagrant community. Eventually, it should probably end up in a Vagrant plug-in, or something similar, but I'm not yet familiar enough with Vagrant's internal to attempt that.

So where's the code already?

I'm getting there, but first a note on terminology. Vagrant and Drupal both use the term "project" to mean very different things, nevermind how the word is used in regular English. Also, Drupal already has a robust templating system that has nothing to do with what I'd been building for Vagrant. So, I prefer the terms "blueprint" instead of "template", and "workspace" instead of "project".

Anyway, here's the current iteration of the Vagrantfile: do |config|
require "./.config/config"

count_vms = 0
Vm.descendants.each do |vm|
count_vms += 1
(1..vm::Count).each do |index|
# Set counters
count = ""
count_fmt = ""
if vm::Count > 1
count = "#{index}"
formatted_count = "(#{index} of #{vm::Count})"

config.vm.define "#{vm::Shortname}#{formatted_count}" do |vm_config| = vm::Basebox
vm_config.vm.box_url = vm::Box_url :hostonly, "#{Conf::Network}.#{Conf::Subnet}.#{Conf::Host_IP + (count_vms * 10) + index - 1}"
hostname = "#{Conf::Workspace}#{count}.#{vm::Domain}"
vm_config.vm.host_name = $hostname
vm_config.vm.customize ["modifyvm", :id, "--name", "#{vm::Longname}#{formatted_count}"]
vm_config.vm.customize ["modifyvm", :id, "--memory", "#{vm::Memory}"]
if defined?(vm::NFS_shares)
vm::NFS_shares.each do |name,path|
vm_config.vm.share_folder("#{name}", "#{path}", "./#{name}", { :nfs => true, :create => true, :remount => true })
if vm::Gui == true
vm_config.vm.boot_mode = :gui

if File::exists?("#{vm::Manifests}/#{vm::Shortname}.pp")
vm_config.vm.provision :puppet do |puppet|
puppet.manifest_file = "#{vm::Shortname}.pp"
puppet.module_path = [ "#{vm::Modules}" , "#{Conf::Modules}" ]
puppet.facter = [
# Fix for broken $fqdn fact on Debian
[ "fqdn", $hostname ],
# Add user variables to be processed in aegir-up::user
[ "aegir_up_username", "#{AegirUpUser::Username}" ],
[ "aegir_up_git_name", "#{AegirUpUser::Git_name}" ],
[ "aegir_up_git_email", "#{AegirUpUser::Git_email}" ],
# Match aegir's uid/gid to the host user for NFS
[ "aegir_user_uid", "#{AegirUpUser::Uid}" ],
[ "aegir_user_gid", "#{AegirUpUser::Gid}" ],
# Set Aegir's root directory and username, again for NFS
[ "aegir_root", "#{vm::Aegir_root}" ],
[ "aegir_user", "#{vm::Aegir_user}" ],
# Tell the Aegir Puppet module not to bother with the 'aegir' user
[ "aegir_user_exists", "true"],
puppet.options = vm::Options
if vm::Debug == true
puppet.options = puppet.options + " --debug"
if vm::Verbose == true
puppet.options = puppet.options + " --verbose"




As you can see, it requires a config file that looks something like this:

require "/home/ergonlogic/.drush/aegir-up_git/lib/blueprints/global.rb"
require "./settings.rb"

class Conf Workspace = "utility"
Modules = "/home/ergonlogic/.drush/aegir-up_git/lib/modules" # puppet modules folder name
Subnet = "11" # 192.168.###.0/24 subnet for this network

class AegirUpUser
Username = "ergonlogic"
Uid = "1000"
Gid = "1000"
Git_name = "Christopher Gervais"
Git_email = "[email protected]"

Which itself includes a global config file:

class Vm # default virtual machine settings
def self.descendants
ObjectSpace.each_object(::Class).select {|klass| klass end

Count = 1 # The number of VMs to create
Basebox = "debian-LAMP-2012-03-29" # default basebox
Box_url = ""
Memory = 512 # default VM memory
Domain = "local" # default domain
Manifests = "manifests" # puppet manifests folder name
Modules = "modules" # puppet modules folder name
Gui = false # start VM with GUI?
Verbose = false # make output verbose?
Debug = false # output debug info?
Options = "" # options to pass to Puppet


class Global
Network = "192.168" # Private network address: ###.###.0.0
Host_IP = 0 # Host address: 192.168.0.###

... and a local settings file:

class Util Count = 2
Shortname = "util" # Vagrant name (used for manifest name, e.g., hm.pp)
Longname = "Utility" # VirtualBox name
NFS_shares = [ "aegir_root" => "/var/aegir", ]
Aegir_root = "/var/aegir" # Shared folder(s)
Aegir_user = "aegir" # Shared folder owner in VM

I'm using classes in somewhat strange way, but it serves my purpose. I'm sure there's a better way to do this, but did I mention I don't have a background in CS or programming? Anyway, I don't want to have any logic in these files, so I'm treating classes as just containers for variables, which have the nifty ability to inherit from other such containers. With only a bit of trickery, I can iterate over any number of VM types defined in such a way.

Initializing an Aegir-up workspace involves essentially copying a blueprint which will include, among other things, a settings.rb file, along with Puppet manifests/modules, &c. It then generates a config.rb file, and creates a symlink back to Aegir-up's Vagrantfile. Aegir-up will also currently write a simple entry in /etc/hosts, but I hope to replace that with a more complete local DNS server (built from another blueprint, no less!)

Moving the Facter facts that get injected out to the blueprints is the next step. By the way, I'll claim full credit for getting this Facter-injection feature into Vagrant, as it was my pull request (if not my code) that inspired Mitchell Hashimoto to build it.

Eventually, I'd like to support Chef as well, since that part of the code is fairly isolated. In fact, since Aegir-up is finally becoming a framework, I've suggested moving said framework code over to Drupal's Vagrant or Drush Vagrant projects.

Anyway, I hope this write-up will help those interested in this and related projects understand how Aegir-up evolved to it's current state, and where I hope we can take it. I suppose more explanation of what's going on in the Vagrantfile and config files could be helpful to that end, but this post is long enough as is it. So anything further will have to wait for another one.


I tweeted at you, but I kinda want to retract, as I think I dismissed the templating approach too easily. Your ruby code looks pretty much like mine -- ie. kinda like the ugly baby that we still love :)

My gut was that it didn't look like an intuitive templating language that I'd be inclined to use, but on further consideration, I think that maybe it's the framing. If the PHPtemplate guys had presented their templating system inside-out with the guts out on the table, it may not have struck people in the way that a templating engine needs to impress. I'm unsure where I'm expected to jump that will make my life simpler. Am I changing the vagrantfile? What does the CLI look like? How does extension work?

If you're looking for feedback from the wider community, I'd maybe suggest hiding the complicated logic that we don't need to know, and present some dead-simple examples of basics and how it helps someone slapping together vagrantfiles or working with multiple VM's. I think this'd help your approach to be compared with the native Vagrantfile format (which can be pretty dead-simple itself), showing exactly what situations your approach makes better.

So I'm thinking I should be playing with the actual aegir-up product before I say anything, because THAT -- the drush integration and everything -- sounds REALLY exciting. And no one cares about our ugly code when we've got a kick-ass idea :)

Really excited to mess around with it later!

Also, just wanted to say that reading through your Vagrantfile has tipped me off to a ton of things that I want to look into, so I totally owe you a debt of gratitude! :)