Setting up a proper dev environment with VirtualBox and Vagrant

For awhile now, I've been meaning to experiment with <a href="http://puppetlabs.com">Puppet</a> to get my server (and maybe even desktop!) config management under control. Since I've called in sick today (I guess I'll just have to miss that budget meeting...), I figure this is a fine opportunity to invest in some skill-building.

Note that I'm not a sysadmin or developer by training; but thankfully, I have the privilege to work with several incredibly competent ones at <a href="http://www.koumbit.org">Koumbit Networks</a>.

A few months back I finally bit the bullet, and deleted the last vestiges of Windows from my laptop. This freed up a bunch of disk space, for lack of which I hadn't been running VMs locally. So with 100+ GBs of free space, I jumped into VirtualBox and started building various machines. Most of these are some form of <a href="http://aegirproject.org">Aegir server</a>; either testing installs using the <em>.deb</em> packaging, hacking on some Drupal contrib modules and Aegir itself, or building some sites locally before pushing them out to production servers.

While working locally in sandbox VMs, with snapshots and such has been great, I've started to run into some limitations of my current setup. First off, I'm using VirtualBox OSE, which is the version in the Ubuntu repos, which I'll need to replace with a more modern version. This is both because the newer version allows for simple cloning of VMs, and that <a href="http://vagrantup.com">Vagrant</a> should work better with it.

Secondly, I've been building the VMs by hand, installing from an Ubuntu server disk image, and then using apt-get to pull the various pieces together that I need, along with configuring network interfaces by hand, etc. Now, I'm a big fan of <a href="http://drupal.org/project/drush_make">Drush Make</a> which allows for building Drupal platforms in a fairly simple declarative manner. The principal reason I'm so interested in Puppet is it's promise of applying similar techniques to system configuration. It also allows me to learn from and leverage the experience and expertise of <em>real</em> sysadmins, who make a point of <a href="https://labs.riseup.net/code/projects/sharedpuppetmodules">sharing their puppet modules and manifests</a>.

Finally, the reason this is becoming more pressing is that I've taken on a project to implement multi-server functionality in Koumbit's Aegir, and so I'd like to have a way to build networks of VMs locally to test various configurations. Also, I'd like to be able to use these same techniques in building OpenAtria.com, which I intend to be a fully open reference implementation of my <a href="http://community.aegirproject.org/handbook/strategy/saas-business-model-... of Drupal-based Software-as-a-Service built atop Aegir</a>.

So, let's dive in! First step, install a <a href="https://www.virtualbox.org/wiki/Linux_Downloads">more recent version of VirtualBox</a>.

Hmm... since switching to <a href="deb http://download.virtualbox.org/virtualbox/debian lucid contrib non-free">Awesome Window Manager</a>, I don't get nagged to update to the latest versions of everything every day. So, I better get things up-to-date:
<code>
sudo aptitude update
sudo aptitude safe-upgrade
</code>

Then, as per the <a href="https://www.virtualbox.org/wiki/Linux_Downloads">install instructions</a>, add the Lucid deb repo (in my case 'deb http://download.virtualbox.org/virtualbox/debian lucid contrib non-free') to my <code>/etc/apt/sources.list</code>, add their repo key, update and install:
<code>
wget http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc
sudo apt-key add oracle_vbox.asc
sudo apt-get update
sudo apt-get install virtualbox-4.1
</code>

This automatically removes the VirtualBox OSE that I'd been using. There are a couple kernel modules (?) that seem to have been left over from the OSE version, but these can be removed with <code>sudo apt-get autoremove</code>. And now, the moment of truth: running <code>virtualbox</code>... and everything just works. Well so far so good.

A couple notes: I added my user to the new group that had been created:
<code>sudo usermod -a -G vboxusers [username]</code>
This was the suggestion by VirtualBox, as USB support was throwing an error. It also auto-discovered a buggy disk config on ext4 that could cause problems. I had in fact been finding VirtualBox VMs a bit flaky, requiring a reboot once a day on average (usually when I was presenting a demo on stage, of course). So I enabled the I/O setting it suggested, and we'll see if that resolves the issue.

Having tested that my existing VMs work, it's time to <a href="http://vagrantup.com/docs/getting-started/index.html">get started with Vagrant</a>. I'm a Drupal developer by trade, so <a href="http://vagrantup.com/docs/getting-started/setup/ubuntu.html">installing Ruby and RubyGems</a> is new to me. The instruction were pretty straight-forward though, and there didn't appear to be any snags. Moving on to installing Vagrant itself was even easier: just a single command.

With Vagrant now installed, I want to get started with a VM:<code>vagrant box add lucid32 http://files.vagrantup.com/lucid32.box</code>, which resulted in:
<code>
[vagrant] Creating home directory since it doesn't exist: /home/socrates32/.vagrant.d
[vagrant] Creating home directory since it doesn't exist: /home/socrates32/.vagrant.d/tmp
[vagrant] Creating home directory since it doesn't exist: /home/socrates32/.vagrant.d/boxes
[vagrant] Creating home directory since it doesn't exist: /home/socrates32/.vagrant.d/logs
[vagrant] Downloading with Vagrant::Downloaders::HTTP...
...
<code>

Now, I don't want my VMs to live in my /home, so I moved them to a dedicated partition and symlinked them:
<code>
mv ~/.vagrant.d/* /media/VMs/vagrant/
rm -rf ~/.vagrant.d/
ln -s /media/VMs/vagrant/ ~/.vagrant.d
</code>

Next, I figure I'll probably have lots of VMs defined this way, so I'll create a directory to keep them in, as well as one for my puppet modules and manifests. Then I need a directory for my first VM, which I'll go ahead and launch:
<code>
mkdir /media/VMs/vagrant/projects
mkdir /media/VMs/vagrant/puppet
mkdir /media/VMs/vagrant/projects/test
cd !$ #nifty bash trick: insert the argument to a previous command into the current input line
vagrant init lucid32
vagrant up
</code>

... and it works! We're almost there now, as there was a small warning on starting up the VM:
<code>
[default] Importing base box 'lucid32'...
[default] The guest additions on this VM do not match the install version of
VirtualBox! This may cause things such as forwarded ports, shared
folders, and more to not work properly. If any of those things fail on
this machine, please update the guest additions and repackage the
box.
Guest Additions Version: 4.1.0
VirtualBox Version: 4.1.2
[default] Matching MAC address for NAT networking...
...
</code>

This can be a little tricky if you don't know where to look for it. There's a CD .iso at <code>/usr/share/virtualbox/VBoxGuestAdditions.iso</code> that you'll need to load into your VM. To do this, uncomment this line from your Vagrantfile:
<code>
config.vm.boot_mode = :gui
</code>
... and restart your VM: <code>vagrant reload</code>. Now that you have a GUI, click on `Devices >> CD/DVD Devices >> Choose a virtual CD/DVD disk file...` Navigate to /usr/share/virtualbox/ and select VBoxGuestAdditions.iso. Login to your VM (<code>vagrant ssh</code>), and install the linux headers (<code>sudo apt-get install dkms</code>. Then mount the disk image (<code>sudo mount /dev/cdrom /media/</code>) and run the installer (<code>sudo sh /media/VBoxLinuxAdditions.run</code>). Once it's done, exit the VM, and restart it (<code>vagrant reload</code>).

To save that update for future re-use as a base box run: <code>vagrant package</code>, which then provides you with a `package.box`. I've renamed it to `updated.box` and moved it to my ../vagrant folder, so I can easily use it to build new VMs.

That's it! Now I can happily define new VMs in Vagranfiles and launch them easily from the command line. Next, I'll start exploring Puppet to define these VMs' configuration, and since you can <a href="http://vagrantup.com/docs/multivm.html">run sets of VMs from within the same Vagrantfile</a>, I'll be setting up whole networks this way.