7hunderblog

Get ready for the 7hunder!

Building a Vagrant Box From Start to Finish

Vagrant

The goal of Vagrant is to make it so simple to create a local development environment, you’d never want to do it another way again. With two simple commands you can quickly setup your first vagrant environment and with a third command, be connected into your first vagrant box in under a few minutes.

vagrant init precise32 http://files.vagrantup.com/precise32.box
vagrant up
vagrant ssh # you're in!

This power is amazing and so useful for freelancers who work on many different projects, or companies that need to bring new members of a team up to speed as quickly as possible. With the use of Vagrant, the configuration of the development and the production environment can be mirrored as closely as possible. And errors like “works on my machine” can become a thing of the past.

If you’re already a fan a Vagrant, I’m probably preaching to the choir. For more about what Vagrant can do check out the amazing documentation. Or do what I did and pick up the O’Reilly book by Vagrant’s author, Mitchell Hashimoto, Vagrant: Up and Running and read it. It’s a short but jam packed review of the details you need to get to know Vagrant, inside and out.

Why Build A Box?

There are a bunch of amazing boxes out there available on a site called vagrentbox.es, so why would you want to build your own box?

Maybe you want to add a few extra things to your base (a runtime or two like Julia, Erlang, JVM or Python, etc.) then start this as your new “base”.

Maybe you want your box to have more ram or you need your boxes to more closely mirror production and you are building a ram enriched, multiple server cluster with with multiple provisioners, we get it… you’ve got a mountain to climb, you just need a map.

Well here’s your map, let’s grab just enough rope. :D

Definitions

What is a package.box file? When using the VirtualBox provider, it’s a tarred, gzip file containing the following:

Vagrantfile
box-disk.vmdk
box.ovf
metadata.json

The Vagrantfile has some information that will be merged into your Vagrantfile that is created when you run vagrant init boxname in a folder.

The box-disk.vmdk is the virtual hard disk drive.

The box.ovf defines the virtual hardware for the box.

The metadata.json tells vagrant what provider the box works with.

NOTE: These contents would be different for the VMWare provider, etc. For more about that please refer to the vagrant docs on boxes.

Shout Out

I’m a fan of Vagrant and all things automation, they make my life a lot easier.

First I’d like to give credits to my main sources which are:

The blog article Creating a Custom Box from Scratch by Ryan Skoblenick.

And the book Vagrant: Up and Running by Mitchell Hashimoto.

So why did I write this blog article, if Ryan’s existed? Good question.

Well it only got me 90% of the way there. I had problems getting the Guest Tools installed and configuring the SSH User for the operating system using his article.

So my hope is that this article will improve upon that process and add a voice of sanity to those who need some help getting a box built by hand.

Getting Prepared

If you don’t already have Vagrant and VirtualBox, grab those.

Download Vagrant installer for your operating system.

Download VirtualBox installer for your operating system.

NOTE: Grab the VirtualBox Extension Pack as well. For more about the functionality it provides read here.

Prior to the installation packages, Vagrant was distributed as a RubyGem. Installation packages are now the preferred way to install Vagrant, so you should uninstall the RubyGem version and follow the instructions for your platform. The RubyGem-based installation is still supported for Vagrant 1.0.x, but is deprecated and will not be supported in any future versions.

Page 8, “Vagrant: Up and Running by Mitchell Hashimoto (O’Reilly). Copyright 2013 Mitchell Hashimoto, 978-1-449-33583-0.”

RUBY USERS: If you had Vagrant installed as a rubygem, uninstall it before you install vagrant:

gem uninstall vagrant

Run the Vagrant installer and the VirtualBox and VirtualBox Extension Pack installers, once they are installed, if you feel like a reboot would help, go ahead.

Download ubuntu server, the rest of the instructions will be for ubuntu, but if you’re choosing another distro you’re probably fine to adapt the commands that follow.

Build a Box

1

We’re going to use VirtualBox to build a ubuntu server from scratch. The reason for this is because Vagrant has native support for VirtualBox. There are more plugins out there for other providers like VMWare, Parallels or Vagrant-LXC, etc. We’ll stick to VirtualBox for this guide.

When we setup our ubuntu server, it prompts us to setup a default user. We’re going to name that user vagrant so it’s the default user as well. This will make it the default SSH user and streamline the process.

Configure the Virtual Hardware

Create a new Virtual Machine with the following settings:

  • Name: vagrant-ubuntu64
  • Type: Linux
  • Version: Ubuntu64
  • Memory Size: 512MB (to taste)
  • New Virtual Disk: [Type: VMDK, Size: 40 GB]

Modify the hardware settings of the virtual machine for performance and because SSH needs port-forwarding enabled for the vagrant user:

  • Disable audio
  • Disable USB
  • Ensure Network Adapter 1 is set to NAT
  • Add this port-forwarding rule: [Name: SSH, Protocol: TCP, Host IP: blank, Host Port: 2222, Guest IP: blank, Guest Port: 22]

Mount the Linux Distro ISO and boot up the server.

Install The Operating System

Setting up ubuntu is very simple. Follow the on-screen prompts. And when prompted to setup a user, set the user to vagrant and the password to vagrant. It will give you a passive-aggressive guilt trip about it being a weak sauce password. Don’t let that shake you, be strong and soldier through. :D

Set Root Password

In order to setup the Super User, aka root user, you’ll need to be able to sign in as that user. Since I asked you to make vagrant the default user while installing the Operating System in the last step, these commands should help you set the root password and then sign in as root in order to make the next configuration changes below.

sudo passwd root

This will prompt you to type the password twice, where I’d suggest the password “vagrant”.

Now sign in as the root user in order to Setup the Super User next.

su -

Setup the Super User

Vagrant must be able run sudo commands without a password prompt and if you’re not configuring ubuntu at this point, just make sure that requiretty is disabled for the vagrant user.

The most efficient way I’ve found to setup the vagrant user so it’s able to use sudo without being prompted for a password is to add it to the sudoers list like this:

sudo visudo -f /etc/sudoers.d/vagrant

Anything in the /etc/sudoers.d/* folder is included in the “sudoers” privledges when created by the root user. So that’s why we created this as the root user and in that folder.

With that file open add this to the file then save it and exit.

# add vagrant user
vagrant ALL=(ALL) NOPASSWD:ALL

Now you can test that it works by running a simple command:

sudo pwd

It will return the home folder without prompting you for a password if everything is setup correctly. If you are prompted for a password, something’s out of wack and things won’t work right. This step was CRUCIAL for me, so please ensure this test passes for you.

Updating The Operating System

One of the reasons we are building this box is so we can save time by already being as up to date as the box was built. So let’s get up-to-date first.

sudo apt-get update -y
sudo apt-get upgrade -y

Usually if there are kernel updates you’ll want to reboot the server. So do that.

sudo shutdown -r now

Install the Vagrant Key

The only way that all the vagrant commands will be able to communicate over ssh from the host machine to the guest server is if the guest server has this “insecure vagrant key” installed. It’s called “insecure” because essentially everyone has this same key and anyone can hack into everyone’s vagrant box if you use it.

But at the same time, we’re hoping you’re not running around with all your most valuable company data on your vagrant boxes, right? RIGHT? OK. Good.

mkdir -p /home/vagrant/.ssh
chmod 0700 /home/vagrant/.ssh
wget --no-check-certificate \
    https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub \
    -O /home/vagrant/.ssh/authorized_keys
chmod 0600 /home/vagrant/.ssh/authorized_keys
chown -R vagrant /home/vagrant/.ssh

Install and Configure OpenSSH Server

If you didn’t install SSH while installing the operating system you can do it now:

sudo apt-get install -y openssh-server

We need to edit the /etc/ssh/sshd_config file:

sudo nano /etc/ssh/sshd_config

Find and uncomment the following line because we added the Vagrant key above to the authorized_keys file:

AuthorizedKeysFile %h/.ssh/authorized_keys

Then restart ssh:

sudo service ssh restart

Installing Guest Tools

Guest Tools help the operating system handle shared folders and “optimize the guest operating system for better performance and usability.” 2

A compiler is required to install the Guest Tools, use this command:

sudo apt-get install -y gcc build-essential linux-headers-server

In VirtualBox browse to the Devices menu at the top, then in the drop-down list at the bottom, click on Insert Guest Additions CD Image.

This will add a ISO image to the virtual CDROM running in your server. Run these commands to mount your cdrom and then run the script. NOTE: The message about the cdrom being read-only is fine.

sudo mount /dev/cdrom /mnt 
cd /mnt
sudo ./VBoxLinuxAdditions.run

Package the Box

Before you package the box you’ll want to “zero out” the drive. “This fixes fragmentation issues with the underlying disk, which allows it to compress much more efficiently later.” 3

sudo dd if=/dev/zero of=/EMPTY bs=1M
sudo rm -f /EMPTY

Then now we’re ready to package the box. I usually make a folder to hold my boxes like so:

mkdir ~/code/personal/vagrant_boxes
cd ~/code/personal/vagrant_boxes

This is the command that finally packages up the box for you as we defined above into the compressed gzip tarball file, it also generates and includes the Vagrantfile and the metadata.json file.

vagrant package --base vagrant-ubuntu64

Vagrant will then check VirtualBox for any instances of the name vagrant-ubuntu64 and attempt to ssh into them and control them.

→ vagrant package --base vagrant-ubuntu64
[vagrant-ubuntu64] Attempting graceful shutdown of VM...
[vagrant-ubuntu64] Forcing shutdown of VM...
[vagrant-ubuntu64] Clearing any previously set forwarded ports...
[vagrant-ubuntu64] Exporting VM...
[vagrant-ubuntu64] Compressing package to: /Users/tbird/code/personal/virtual_boxes/package.box

You are left with the package.box file in your ~/code/personal/vagrant_boxes folder.

Test Your Box

From your same vagrant_boxes folder you can run these final test commands. All the heavy lifting is really done at this point. If you’ve screwed up something it’s probably in a step up above. :P

You should be in ~/code/personal/vagrant_boxes/ and type:

vagrant box add ubuntu64 package.box
vagrant init ubuntu64
vagrant up

Connect to your server you created from start to finish!

vagrant ssh

You’ve won, you deserve to get your high five on.

Packer

Just when you thought it was safe to do things manually at the beginning … in comes Packer.

What is Packer, you say? Well Packer automates everything we just did.

Anyway… Once you have Vagrant, VirtualBox, and Packer installed you can define a quick-start.json file like:

{
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "YOUR KEY HERE",
    "secret_key": "YOUR SECRET KEY HERE",
    "region": "us-east-1",
    "source_ami": "ami-de0d9eb7",
    "instance_type": "t1.micro",
    "ssh_username": "ubuntu",
    "ami_name": "packer-example "
  }]
}

After changing the Access and Secret Key you could run the packer command with the quick-start.json file:

packer build quick-start.json

Packer will then automate the creation of that quick-start.json machine image for Amazon EC2. That can optionally include the creation of a Vagrant box.

What Packer aims to do is to make their images compatible with all these providers: Amazon EC2 (AMI), DigitalOcean, Docker, Google Compute Engine, OpenStack, QEMU for KVM or Xen instances, or even VirtualBox or VMWare software. (See platforms for the entire list.)

What Have We Learned

What have we learned from this process of building boxes for Vagrant?

With great power comes great use of cliche phrases!

The process of automation is the process of taking yourself out of the equation. “Auto” is defined as self, and the suffix “ation” is defined as action. So if we think of the job of automating your development environment or production environment so it runs without your intervention, then your actions are doing it right!

And even if we want to do it by hand or use Packer, knowing how things work is a lot better than guessing how they work. And I like to experience things a few times before I start to automate those boring parts away, and then focus on new more interesting challenges.

Footnotes

  1. http://giphy.com/gifs/povenlBAIz14s
  2. VirtualBox Manual - Chapter 4: Guest Additions
  3. Page 81, “Vagrant: Up and Running by Mitchell Hashimoto (O’Reilly). Copyright 2013 Mitchell Hashimoto, 978-1-449-33583-0.”

Integrate Two-Factor GitHub With RubyMine

Security and Integrity

So you want to have both Security and Integrity. You want to keep your two factor authentication from GitHub and you want your RubyMine version control to authenticate.

You’ll need to setup an access token for RubyMine and then configure your IDE to use it.

Github

Login to Gitub and create a token.

RubyMine

Update RubyMine to use your token.

Enjoy!

Build a Chef Solo Development Environment With Vagrant

Shared Folder Demo

local$
ls
echo "<h1>Hello from a Vagrant VM</h1>" > index.html
vagrant ssh

vagrant$
ls /vagrant
wget -qO- 127.0.0.1

BROWSER
http://localhost:4567

local$ 
cd ~/code/personal/chef_projects/demo/
mkdir -p chef/cookbooks/main/recipes/
touch chef/cookbooks/main/recipes/default.rb
touch solo.rb
touch node.json
subl .

Goodbye Dear Friend

Tonight marks the passing of a computer that I’ve had for 10 years!

10 years ago there was a big schism in the RAM architecture world. And when I was building my machine I got caught in the middle of this debate. Should I go with DIMM or RIMM memory? What was better technology? At that particular moment, DIMM’s main selling point (if I even remember correctly) was price. And that’s ultimately how they won. Like the argument of VHS versus Beta, Beta may have been the better technology, but VHS made it to market cheaper. Probably because Sony was being proprietary dorks about Beta, but that’s not what this is about…

Yet I chose RIMMs, why? Because you don’t know the POWER of the dark side! In hind sight I backed the wrong horse. Because even though in 2003 1 GB of ram seemed like a lot, eventually when I wanted to upgrade later, the costs to upgrade were so prohibitive it would have been cheaper to replace the motherboard and CPU, than upgrade to 2GB of ram.

Ironically it was that 2GB upgrade that was the final nail in the coffin of our tale, but we’ll come back to this…

Over the years that machine has seen every configuration and upgrade possible. I’ve had multiple optical burners, a number of hard drive configurations from single system to RAID arrays. I’ve upgraded the video card 3 different times, and even on an AGP 8x slot no less! The final card being one that could actually run modern flash sites like Hulu decently (paired with a CPU upgrade as well.)

And through it all the motherboard, the original GIGABYTE GA-8IHXP rev 2.1 has been the champion. I even eventually renamed this computer “Blue Phoenix” because it has risen from the ashes of so many re-configurations that it seemed unbreakable. (Blue was from the color of case at the time.)

This computer even survived two cross-country moves from Utah to Florida, then Florida back to Utah, and still worked fine. The only problem I had when we got back from Florida was the case had got banged up by my Aluminum Mac tower so bad, that it was really scratched up. So I decided to buy a new case and thus, the phoenix rose from the ashes, reborn again as the “Dark Phoenix” in it’s new black case.

I’ve got other computers I use now on a daily basis. Like the one I’m typing this blog post on, my iMac. But I still had uses for the “Dark Phoenix”. As I mentioned before, I had upgraded the CPU and with that upgrade I was able to have a better response with running video capture card and turning the computer into a HTPC (Home Theater Personal Computer).

Basically it could serve as a DVR (Digital Video Recorder) and record TV for us, and we could watch anything recorded on the Xbox. Or we could pause live TV. It was great!

Then my mom’s computer died on her and she needed to have a computer, until she got a replacement. So I offered her my Dark Phoenix box as a temporary replacement. She has had it for the past few months and then we just replaced it with a Macbook Air for her.

That’s when I brought the Dark Phoenix home, and had bought 2 GB of ram to put into the machine. Time had finally caught up where the RIMM prices were low enough that I could justify the upgrade. And so I began to try and put the ram in.

At first it wouldn’t boot up, so I replaced the old ram back in and tried to reboot and it came back up, but not consistently.

The next day, while I was on the phone with someone from work, a loud POP noise came from the direction of the computer. I checked to see if there were any suspicious “electronics smells” that usually accompany a bad frying of circuits, but there was nothing.

So this weekend I tore down the entire machine and cleaned everything. I built it back up to the bare minimums, using only the old ram (which should have worked because it was working before). And I even tried replacing the video card (part of a minimum boot up) with an old video card. And still… no luck.

It is with profound sadness that I must bid farewell to my friend the Dark Phoenix. To me, at least, psychologically, if I have to replace the motherboard, the machine just isn’t the same and is a new computer. The motherboard is the “heart and soul”, and all other components are just upgrades.

RIP Dark Phoenix. 8/3/2002 - 1/22/2012

Host Hudson CI on Engine Yard AppCloud

Hudson CI

The Hudson Continuous Integration server, built on Java, has been adopted by Dr. Nic. He has pioneered the engineyard-hudson project that helps grease the wheels and get a Hudson server running on Engine Yard’s AppCloud in just minutes.

Dr. Nic recorded the screencast footage and then I assisted by providing my best “radio voice” for the final voice-over and mixed in our latest screencast mojo.

REVISION: Hudson became Jenkins, check out the blog article on how to Boot Up Your Own Dedicated Jenkins CI in Two Commands and https://github.com/engineyard/eycloud-app-jenkins/ is the updated repository.

Engine Yard Is Out of the Hardware Business

Fun With Sledgehammers

While I was recently in San Francisco we had a company meeting and celebrated a huge milestone. We had recently completed an almost year-long migration from our original legacy hardware to our new cloud based virtual machine offerings.

And so Tom Mornini and Ed Muller were able to take a swing at some of the original hardware with a Sledgehammer.

Amazon Out of Capacity

Quickcasts

To answer an answer quickly, and in screencast form, I created the Amazon Out of Capacity Quickcast and published it.

The most popular size EY AppCloud offers is the High CPU Medium. Sometime this size will not be available at the time a user would like to boot an instance. So this video outlines a simple step you can take to get the instance to boot, even when demand is high.