Aegir-up: a Vagrant project templating framework

!!! WARNING: This blog post contains code !!!

Quite a bit of it, actually. It's in Ruby. I've never coded in Ruby before this, so in all likelihood it sucks. But I probably won't be able to recognize that until I've had more experience hacking on Ruby code (maybe in Vagrant or Puppet). Also, my academic and professional background is in philosophy and marketing, so there's no help there. Anyway, consider yourself warned. That said, I'm very open to constructive criticism, so feel free to make suggestions.

So, what's this all about then?

I discovered Vagrant about 6 months ago, when a friend and colleague suggested I try it out, as a way to learn Puppet. I fell in love with both technologies. Just like when I discovered Drupal (~5 years ago), I was able to start using them productively almost immediately, but I also found an easy on-ramp to delving deeper.

I'd wanted to learn Puppet at the time in order to further automate deployment and management of Aegir servers, as we ramped up Koumbit's AegirVPS service. I also wanted to be able to test things like mass migration of sites between Aegir servers, and Vagrant provided a perfect tool to do both.

Once the Aegir Puppet module started to come together, it became clear that deploying an Aegir server locally, within Vagrant, was worthwhile in it's own right. And so, Aegir-up was born.

Initially, I found it challenging to find the relevant bits in a Vagrantfile to allow me to alter settings. So I began to abstract my Vagrantfile, and isolate all settings in an included 'settings.rb' file. This resulted in an .ini-like control file. Then I started getting frustrated as I'd forget which settings had to be tweaked with each new project, to avoid collisions and such. So, I started to learn Bash scripting in order to automate project initialization.

Automating project creation implied the use of a template, which then begged to be further generalized. So, I ended up with a template that would deploy an Aegir server from the Debian packages, and another that would install everything "manually" directly from the Git repositories. Along the way, I discovered Veewee, and started bundling customized base box definitions within Aegir-up.

At this point, things started to get a little out-of-hand. The scripts ended up with bits of perl, sed and other bits of Unix tools embedded within them, along with duplicated argument handling code, and such. Seeing as how this was essentially a Drupal-specific tool, I heeded the advice of one of Aegir's maintainers, Steven Jones, and re-wrote the scripts as a Drush extension. this was a great opportunity to further delve into the intricacies of Drush, since most of the heavy lifting in Aegir is done in a Drush extension, Provision.

Steven also pointed out that the most powerful piece of Aegir-up was this templating system, that could be generalized into a framework. Lately, I've been working on just that "frameworkifying", and it's almost there. I'm now considering adding a Rake script to handle some of the template work, and just calling that script from within Drush.This would have the benefit of making the template framework independent of Drush, and thus potentially useful for the broader Vagrant community. Eventually, it should probably end up in a Vagrant plug-in, or something similar, but I'm not yet familiar enough with Vagrant's internal to attempt that.

So where's the code already?

I'm getting there, but first a note on terminology. Vagrant and Drupal both use the term "project" to mean very different things, nevermind how the word is used in regular English. Also, Drupal already has a robust templating system that has nothing to do with what I'd been building for Vagrant. So, I prefer the terms "blueprint" instead of "template", and "workspace" instead of "project".

Anyway, here's the current iteration of the Vagrantfile:

Vagrant::Config.run do |config|
  require "./.config/config"
 
  count_vms = 0
  Vm.descendants.each do |vm|
    count_vms += 1
    (1..vm::Count).each do |index|
      # Set counters
      count = ""
      count_fmt = ""
      if vm::Count > 1
        count = "#{index}"
        formatted_count = "(#{index} of #{vm::Count})"
      end
 
      config.vm.define "#{vm::Shortname}#{formatted_count}" do |vm_config|
 
        vm_config.vm.box = vm::Basebox
        vm_config.vm.box_url = vm::Box_url
        vm_config.vm.network :hostonly, "#{Conf::Network}.#{Conf::Subnet}.#{Conf::Host_IP + (count_vms * 10) + index - 1}"
        hostname = "#{Conf::Workspace}#{count}.#{vm::Domain}"
        vm_config.vm.host_name = $hostname
        vm_config.vm.customize ["modifyvm", :id, "--name", "#{vm::Longname}#{formatted_count}"]
        vm_config.vm.customize ["modifyvm", :id, "--memory", "#{vm::Memory}"]
        if defined?(vm::NFS_shares)
          vm::NFS_shares.each do |name,path|
            vm_config.vm.share_folder("#{name}", "#{path}", "./#{name}", { :nfs => true, :create => true, :remount => true })
          end
        end
        if vm::Gui == true
          vm_config.vm.boot_mode = :gui
        end
 
        if File::exists?("#{vm::Manifests}/#{vm::Shortname}.pp")
          vm_config.vm.provision :puppet do |puppet|
            puppet.manifest_file = "#{vm::Shortname}.pp"
            puppet.module_path   = [ "#{vm::Modules}" , "#{Conf::Modules}" ]
            puppet.facter = [
              # Fix for broken $fqdn fact on Debian
              [ "fqdn",               $hostname ],
              # Add user variables to be processed in aegir-up::user
              [ "aegir_up_username",  "#{AegirUpUser::Username}" ],
              [ "aegir_up_git_name",  "#{AegirUpUser::Git_name}" ],
              [ "aegir_up_git_email", "#{AegirUpUser::Git_email}" ],
              # Match aegir's uid/gid to the host user for NFS
              [ "aegir_user_uid",     "#{AegirUpUser::Uid}" ],
              [ "aegir_user_gid",     "#{AegirUpUser::Gid}" ],
              # Set Aegir's root directory and username, again for NFS
              [ "aegir_root",         "#{vm::Aegir_root}" ],
              [ "aegir_user",         "#{vm::Aegir_user}" ],
              # Tell the Aegir Puppet module not to bother with the 'aegir' user 
              [ "aegir_user_exists",  "true"],
            ]
            puppet.options = vm::Options
            if vm::Debug == true
              puppet.options = puppet.options + " --debug"
            end
            if vm::Verbose == true
              puppet.options = puppet.options + " --verbose"
            end
          end
        end
 
      end
 
    end
  end
 
end

As you can see, it requires a config file that looks something like this:

require "/home/ergonlogic/.drush/aegir-up_git/lib/blueprints/global.rb"
require "./settings.rb"
 
class Conf < Global
  Workspace = "utility"
  Modules   = "/home/ergonlogic/.drush/aegir-up_git/lib/modules"   # puppet modules folder name
  Subnet    = "11"                  # 192.168.###.0/24 subnet for this network
end
 
class AegirUpUser
  Username  = "ergonlogic"
  Uid       = "1000"
  Gid       = "1000"
  Git_name  = "Christopher Gervais"
  Git_email = "chris@ergonlogic.com"
end

Which itself includes a global config file:

class Vm                                 # default virtual machine settings
  def self.descendants
    ObjectSpace.each_object(::Class).select {|klass| klass < self }
  end
 
  Count     = 1                          # The number of VMs to create
  Basebox   = "debian-LAMP-2012-03-29"   # default basebox
  Box_url   = "http://ergonlogic.com/files/boxes/debian-LAMP-current.box"
  Memory    = 512                        # default VM memory
  Domain    = "local"                    # default domain
  Manifests = "manifests"                # puppet manifests folder name
  Modules   = "modules"                  # puppet modules folder name
  Gui       = false                      # start VM with GUI?
  Verbose   = false                      # make output verbose?
  Debug     = false                      # output debug info?
  Options   = ""                         # options to pass to Puppet
 
end
 
class Global
  Network   = "192.168"                  # Private network address: ###.###.0.0
  Host_IP   = 0                          # Host address: 192.168.0.###
end

... and a local settings file:

class Util < Vm             # VM-specific overrides of default settings
  Count      = 2
  Shortname  = "util"          # Vagrant name (used for manifest name, e.g., hm.pp)
  Longname   = "Utility"       # VirtualBox name
  NFS_shares = [ "aegir_root" => "/var/aegir", ]
  Aegir_root = "/var/aegir"    # Shared folder(s)
  Aegir_user = "aegir"         # Shared folder owner in VM
end

I'm using classes in somewhat strange way, but it serves my purpose. I'm sure there's a better way to do this, but did I mention I don't have a background in CS or programming? Anyway, I don't want to have any logic in these files, so I'm treating classes as just containers for variables, which have the nifty ability to inherit from other such containers. With only a bit of trickery, I can iterate over any number of VM types defined in such a way.

Initializing an Aegir-up workspace involves essentially copying a blueprint which will include, among other things, a settings.rb file, along with Puppet manifests/modules, &c. It then generates a config.rb file, and creates a symlink back to Aegir-up's Vagrantfile. Aegir-up will also currently write a simple entry in /etc/hosts, but I hope to replace that with a more complete local DNS server (built from another blueprint, no less!)

Moving the Facter facts that get injected out to the blueprints is the next step. By the way, I'll claim full credit for getting this Facter-injection feature into Vagrant, as it was my pull request (if not my code) that inspired Mitchell Hashimoto to build it.

Eventually, I'd like to support Chef as well, since that part of the code is fairly isolated. In fact, since Aegir-up is finally becoming a framework, I've suggested moving said framework code over to Drupal's Vagrant or Drush Vagrant projects.

Anyway, I hope this write-up will help those interested in this and related projects understand how Aegir-up evolved to it's current state, and where I hope we can take it. I suppose more explanation of what's going on in the Vagrantfile and config files could be helpful to that end, but this post is long enough as is it. So anything further will have to wait for another one.

Comments

I tweeted at you, but I kinda want to retract, as I think I dismissed the templating approach too easily. Your ruby code looks pretty much like mine -- ie. kinda like the ugly baby that we still love :)

My gut was that it didn't look like an intuitive templating language that I'd be inclined to use, but on further consideration, I think that maybe it's the framing. If the PHPtemplate guys had presented their templating system inside-out with the guts out on the table, it may not have struck people in the way that a templating engine needs to impress. I'm unsure where I'm expected to jump that will make my life simpler. Am I changing the vagrantfile? What does the CLI look like? How does extension work?

If you're looking for feedback from the wider community, I'd maybe suggest hiding the complicated logic that we don't need to know, and present some dead-simple examples of basics and how it helps someone slapping together vagrantfiles or working with multiple VM's. I think this'd help your approach to be compared with the native Vagrantfile format (which can be pretty dead-simple itself), showing exactly what situations your approach makes better.

So I'm thinking I should be playing with the actual aegir-up product before I say anything, because THAT -- the drush integration and everything -- sounds REALLY exciting. And no one cares about our ugly code when we've got a kick-ass idea :)

Really excited to mess around with it later!

Also, just wanted to say that reading through your Vagrantfile has tipped me off to a ton of things that I want to look into, so I totally owe you a debt of gratitude! :)