Boxen: OSX Configuration Beyond Dotfiles

Jim Moore & Brian Street

Abstract

“Infrastructure as code” is accepted for our servers, but why don’t we apply it to our desktops? Boxen is a tool that enables not just automated installation of software, but also configuration of our desktops. Boxen puts the developer in control of their desktop (as if they weren’t anyway) but also scales across companies, teams, and even home vs work computers.

We will cover the basic ideas and tools involved, as well showing some of the things you can configure (hint: everything) for really pimping your machine and making upgrading your environment much faster and consistent WAAAY beyond dotfiles. We’ll also talk about our experiences in scaling this across the developers in our team.

Introducing Jim

Jim Moore

Jim Moore (jdigger jdiggerj)

Introducing Brian

GitHub Avatar

Brian Street (ItGumby ItGumby)

Introduction: Summary

Collectively, we work on a development team with more contractors than employees.

We have had to “spin-up” quite a few people.

We also had a batch of hot new laptops for some employees…

Why Boxen: Consistency & Repeatability

Why Boxen: Faster

Why Boxen: Cool

Why Boxen: Desktops are “Special”

How do you automate software installation on a Mac (or any “desktop” OS)?

Requirements

Primary Components

GitHub

version control and user management

Puppet

system configuration

Puppet Librarian

module management for Puppet

Homebrew

provides “most” of the software

Ruby (rbenv)

most of the scripting

Software Package Providers

There are many “providers” of software. Some examples:

Provider Notes

homebrew

the default for Boxen, it’s one of the most fully capable

appdmg

if *.dmg contains *.app to drag into /Applications

appdmg_eula

if *.dmg asks for a license agreement when it is opened and contains *.app to drag into /Applications

pkgdmg

if *.pkg contains installer

compressed_app

if *.zip to unzip & place into /Applications

macports

supported, but doesn’t always play well with homebrew

Other sources include gem, npm, pip, …

Provider Features

The OS X packaging providers are, in general, incredibly “limited”. By definition, providers install software. Some can do more:

Provider Uninstall Versionable Updatable

appdmg

 

 

 

appdmg_eula

 

 

 

compressed_app

 

 

compressed_pkg

 

 

 

homebrew

macports

pkgdmg

 

 

 

Providers (other than homebrew & macports) track package installation at /var/db/.puppet_{provider}_installed_{package_name}

Therefore, to re-install, you must delete the file above before running boxen again.

Boxen Limitations

The primary limitation is that it’s not (yet?) able to install software from the Apple App Store.

Otherwise install and configure your entire system automatically without manual steps. The time cost is primarily download time.

Getting Started: The Bootstrap

sudo mkdir -p /opt/boxen
sudo chown ${USER}:staff /opt/boxen
git clone [your_boxen_repo] /opt/boxen/repo
cd /opt/boxen/repo
./script/boxen [--no-fde](1)
1 defaults to Full Disk Encryption

Alternate: Boxen Web

  • boxen-web

    • uses heroku to provide customized homepage & OAuth

Getting Started: Customize & Iterate

Puppet Terms: Basic Structure

(Highly, highly simplified)

Module

Effectively Puppet’s “library” unit

Manifest

The file containing the description/script

Resource

A unit of configuration ( e.g., “file”, “package” )

Class

A singleton managing Resources

Defined Type

Can have more than one instance managing Resources

Puppet Glossary

In general, “Module” has at least one “Manifest” which has at least one “Class”

Puppet Terms: Resources

Every resource has:

Resources can (and often do) depend on other resources. Puppet will build a dependency graph to make sure everything is applied in the correct order. Each resource title must be unique in the DAG.

Puppet Terms: Misc

Facts (Facter)

Descrete information about the machine (FQDN, IP addr, OS, etc.) gathered by “facter”

Puppet Librarian

Used for managing Puppet Modules

Hiera

Hierarchical data: a flexible way of providing configuration data based on “facts”

Profiles & Roles

Not covered here, but a Puppet Enterprise convention for organizing Hiera data

Boxen Terms

User

Defined by the “fact”: ::boxen_user

Project

A grouping meant for “make sure people on this team have at least this configuration”; you can have multiple “projects” applied to a machine

Packages

Packages are the primary way software is installed.

Simple example:

package { 'gradle':
    ensure   => "installed",(1)
    provider => "homebrew",
}

package { "IntelliJ-IC-12.1.4":
    provider => 'appdmg_eula',
    source   => "http://download.jetbrains.com/idea/ideaIC-12.1.4.dmg",
}
1 typically defaults to “present”. Valid values vary by resource type & provider. May include installed, latest, absent, or a specific version/version range (e.g., ">= 1.12")

Other Resources: File

Puppet File Documentation

$home = "/Users/${::boxen_user}"(1)

file { "${home}/.zshrc":
  source => 'puppet:///modules/people/jdigger/zshrc',(2)
}

file { "${home}/.zshenv":
  content => template('people/jdigger/zshenv.erb'),(3)
}
1 define a variable based on a “fact”
2 a static file from a module
3 set the content from a template (Ruby ERB by default)

Other Resources: Repository

GitHub Repository

$home = "/Users/${::boxen_user}"
$srcdir = "${home}/src"

repository { "${srcdir}/git-process" :
  source   => 'https://github.com/jdigger/git-process.git',
  path     => "${srcdir}/git-process",
  provider => 'git',
}

Other Resources: Ruby Gem

$ruby_version = '1.9.3'

ruby::gem { "git-process for ${ruby_version}":(1)
    gem     => 'git-process',
    ruby    => $ruby_version,(2)
    version => '~> 2.0',(3)
}
1 because there could be potentially more than one installation of the gem (in the various rbenv versions) it’s a good idea to put the Ruby version in the resource name so it is guaranteed to be unique
2 the version of Ruby (using rbenv) to install the gem into
3 of course gem versions can use the semantic versioning support of Ruby Gems

Other Resources: Plist

example of configuring Adium
property_list_key { 'Adium users':
  path       => "${home}/Library/Application Support/Adium 2.0/Users/Default/Accounts.plist",
  key        => 'Accounts',
  value      => [
    {
      'Service'  => 'GTalk',
      'UID'      => $gtalk_name,
      'Type'     => 'libpurple-jabber-gtalk',
      'ObjectID' => '1',
    },
    {
      'Service'  => 'Yahoo!',
      'UID'      => $yahoo_name,
      'Type'     => 'libpurple-Yahoo!',
      'ObjectID' => '2',
    },
    {
      'Service'  => 'AIM',
      'UID'      => $aim_name,
      'Type'     => 'libpurple-oscar-AIM',
      'ObjectID' => '3',
    },
  ],
  value_type => 'array',
}

Requires the “glarizza/puppet-property_list_key” module

See the plutil man page.

Other Resources: OSX

boxen::osx_defaults { 'scrollbars always on':
  domain => 'NSGlobalDomain',
  key    => 'AppleShowScrollBars',
  value  => 'Always',
  user   => $::boxen_user,
}
osx::recovery_message { 'If this Mac is found, please call 555-555-5555': }
include osx::finder::unhide_library

See the defaults man page.

An awesome list of available settings/tools can be found at OSXDefaults.

Other Resources: Exec

Puppet Exec Documentation

$source_tgz =
    'http://closure-linter.googlecode.com/files/closure_linter-latest.tar.gz'(1)

exec { 'install gjslint':(2)
    command => "easy_install ${source_tgz}",
    user    => 'root',(3)
    creates => '/usr/local/bin/gjslint',(4)
}
1 should come in as part of a class definition…
2 arbitrary resource name: if “command” not given, this is used, but generally best to give it a “meaningful” name
3 the user to sudo as when executing the command
4 before running the command the existence of this file is checked; if it’s there it’s assume this has already run

Instead of “creates”, you can use “onlyif”/“unless” to run a command, such as

unless => 'grep root /usr/lib/cron/cron.allow 2>/dev/null'

Defining Classes

The primary way of referencing groups of Resources is via Classes.

A Class may take parameters for configuration.

from puppet-intellij (slide-ware version)
class intellij($edition='community', $version='13.1.1') {
  case $edition {
    'community': { $edition_real = 'IC' }
    'ultimate': { $edition_real = 'IU' }
    default: { fail('Class[intellij]: parameter edition must be community or ultimate') }
  }

  package { "IntelliJ-IDEA-${edition_real}-${version}":
    provider => 'appdmg_eula',
    source   => "http://download.jetbrains.com/idea/idea${edition_real}-${version}.dmg",
  }
}

Defining Classes (annotated)

class intellij($edition='community', $version='13.1.1') {
  case $edition { (1)
    'community': { $edition_real = 'IC' }
    'ultimate': { $edition_real = 'IU' }
    default: { fail('Class[intellij]: parameter edition must be community or ultimate') } (2)
  }

  package { "IntelliJ-IDEA-${edition_real}-${version}": (3)
    provider => 'appdmg_eula', (4)
    source   => "http://download.jetbrains.com/idea/idea${edition_real}-${version}.dmg", (5)
  }
}
1 Conditionals for setting "private" variable
2 Able to fail the configuration before anything is applied
3 Variable substitution. (Resource names must be unique)
4 Install from .dmg, auto-accepting the EULA
5 Where to download the installation package from

Calling Classes

Explicitly calling with parameters
class { "intellij":
    edition => 'community',
    version => '13.1.1',
}
Calling with default parameters
class { "intellij": }
Importing with default parameters
import "intellij"

The import form has the advantage over class {:} in that class {:} can only appear once in your entire graph. import will add the class resource if it’s not yet defined, or ignore it if not.

Calling Classes with Hiera

Want simple resource declarations AND centralized/powerful configuration data (like versions)? That’s what Hiera is for…

Importing with parameter lookup
import "intellij"
Hiera YAML configuration
intellij::edition: 'ultimate'
intellij::version: '13.1.2'

Separates usage from configuration

Defined Types

Defined Types are similar to Classes, providing scoping, resource management, ability to pass in parameters, etc.

Unlike Classes, you can have multiple instances. Because of that, you can’t use include or tools like Hiera to configure them.

from the boxen/puppet-sublime_text_2 modules
define sublime_text_2::package($source) {
  require sublime_text_2::config

  repository { "${sublime_text_2::config::packagedir}/${name}":
    source => $source
  }
}

If you’re trying to decide between declaring something class or define, err toward class

Resource Defaults

repo/manifests/site.pp (simplified)
Exec {
  group => 'staff',
  user  => $boxen_user,
  path  => [
    "${boxen::config::home}/rbenv/shims",
    "${boxen::config::home}/rbenv/bin",
    "${boxen::config::home}/homebrew/bin",
    '/usr/bin', '/bin', '/usr/sbin', '/sbin',
  ],
}

File {
  group => 'staff',
  owner => $boxen_user,
}

Package {
  provider => homebrew,
}

Repository {
  provider => git,
}

Service {
  provider => ghlaunchd,
}

Brief Structure

Boxen default installation at /opt/boxen

All user edits to the repo subdirectory (/opt/boxen/repo/)

Editing /opt/boxen/repo

Ordered roughly by likelihood that you’ll modify something.

modules/

the Puppet “modules” to load; most customization happens here

modules/people

contains the module associated with a specific person

Puppetfile

control file for Puppet Librarian

hiera/

data-based configuration

modules/projects

contains the module associated with projects

manifests/site.pp

sets the defaults for the loaded modules

config/

contains some files for tweeking more advanced features

Parts to Avoid Editing in /opt/boxen/repo

Directories you should not directly interact with:

script/

contains the primary scripts, including boxen itself

shared/, vendor/

cache for Librarian modules

bin/

the shims for installed Ruby Gems, including Puppet

../

where Boxen installs a lot of its other control files

*.lock

control files for version locking

The *.lock files are great when you understand them, and generally stay out of your way even when you don’t. But they can be a bit of a pain when you merge in remote changes to the project… See the FAQ.

Boxen User Edits

Directory: /opt/boxen/repo/modules/people

By default {user} is based on your GitHub user ID. However, you can leverage parts and pieces from other users.

manifests/{user}.pp

primary manifest for a user

manifests/{user}/

where your “subclasses” go

files/{user}/

static resources for the user

templates/{user}/

templatized resources for the user

spec/

unit tests for the manifests

Example User Manifest

modules/people/jdigger.pp
class people::jdigger {
    include people::jdigger::dotfiles
    include people::jdigger::bin
    include people::jdigger::applications
    include people::jdigger::ruby
    include people::jdigger::git
    include people::jdigger::sublime_text_2
    include people::jdigger::osx
}

Just as it’s good practice in OOD to delegate low-level details to subclasses, you should do the same in Puppet class design as well.

Example “Sub Class”

modules/people/jdigger/sublime_text_2.pp
class people::jdigger::sublime_text_2 {
  include 'sublime_text_2'(1)

  $home = "/Users/${::boxen_user}"

  file { "${home}/Library/Application Support/Sublime Text 2/Packages/User":
    ensure => 'directory',
    owner  => $::boxen_user,
    mode   => '0755',
  } (2)
  -> (3)
  file { "${home}/Library/Application Support/Sublime Text 2/Packages/User/Preferences.sublime-settings":
    source  => 'puppet:///modules/people/jdigger/sublime-settings',
  } (4)
}
1 make use of the “general” sublime_text_2 module
2 ensure that the user preferences directory exists
3 shorthand to declare that the following resource needs the previous one to be applied first
4 I want Sublime Text 2 to behave the same for me regardless of what machine I’m on, so any changes I make to the settings are done in the module

Puppet Librarian: Puppetfile

Puppet Librarian usage/syntax

some standard entries
github "dnsmasq",  "1.0.1"
github "gcc",      "2.0.100"
github "git",      "2.3.0"
github "homebrew", "1.6.2"

Where github is a function that translates

github "git", "2.3.0"

to

mod "git", "2.3.0", :github_tarball => "boxen/puppet-git"

Puppet Librarian: Finding Modules

Search for Module Names

Boxen Facter Extensions

snippet from shared/lib/facter/boxen.rb
dot_boxen   = "#{ENV['HOME']}/.boxen"
user_config = "#{dot_boxen}/config.json"

require "boxen/config"
config = Boxen::Config.load

facts["github_login"]  = config.login
facts["github_email"]  = config.email
facts["github_name"]   = config.name
facts["github_token"]  = config.token

facts["boxen_home"]     = config.homedir
facts["boxen_srcdir"]   = config.srcdir
facts["boxen_repodir"]  = config.repodir
facts["boxen_reponame"] = config.reponame
facts["boxen_user"]     = config.user
#..

The “cached” values for those are in /opt/boxen/config/boxen/defaults.json

You can set your own personal/private custom Facts in ~/.boxen/config.json

Use Case: Private vs Personal

Typical personal configuration goes in modules/${::github_login}, which is public to a boxen repo. This is generally a very good thing.

However, some facts do not belong in a public location (Passwords, SSH keys, OAuth tokens, etc.)

For configuration that can easily be turned into “data” — especially if it would be used in multiple places when configuring the system (e.g., user names and passwords) — the ~/.boxen/config.json file is perfect.

For configuration that’s more complex, like your ~/.ssh directory, create a private BitBucket repository, or encrypted .zip in Dropbox, or …

Keep it simple and secure: You want it to be easily accessible before the rest of your system is set up.

Tips: General

Tips: Infrastructure As Code

Don’t forget it’s all code and backed by git

Use the forking and branching processes you normally would

merge with the upstream (e.g., boxen/our-boxen) often
git remote add boxen https://github.com/boxen/our-boxen.git
git fetch --all
git merge boxen/master

When you get merge conflicts on *.lock files, see the FAQ

Make use of other people’s modules, and post your own!

Tips: Re-use by Module

Create modules/packages/ for general-purpose Puppet classes to share across users or projects

Publish useful packages. OSS benefits all, and the “packages” module and its like should be considered temporary/stop-gap.

This also works well for “proprietary” software/configurations (though those may be published on an internal repo instead of a public one).

Tips: Avoid ‘node default’

The default repo/manifests/site.pp contains a node default section that will get loaded up on every machine. There’s two problems with it:

  1. The “out of the box” list contains a bunch of stuff that non-GitHub.com people may or may not care about having loaded up on their machine (e.g. NodeJS, ngnix, 4 versions of Ruby)
  2. node default doesn’t allow changing configuration values, etc.

It’s a legacy of Puppet 1.0, long before much more flexible mechanisms like Hiera.

Tips: Use Hiera

A much better approach

replace ‘node default’ in ‘site.pp’ with
if hiera_array('classes', undef) {
  hiera_include('classes')
}

Then, if you want to make sure everyone’s got Ruby set up…

hiera/common.yaml
classes:
  - ruby::global
ruby::global::version: "2.1.1"
ruby::rbenv_plugins:
  ruby-build:
    ensure: v20140420
    source: sstephenson/ruby-build

Tips: Hiera Power

With a simple shim you can do things like this:

hiera/developer.yaml
sublime_text_2::packages:
  'BracketHighlighter':
    source: 'facelessuser/BracketHighlighter'
  'CodeIntel':
    source: 'SublimeCodeIntel/SublimeCodeIntel'
  'Git':
    source: 'kemayo/sublime-text-git'
  'GitGutter':
    source: 'jisaacks/GitGutter'
hiera/users/jdigger.yaml
sublime_text_2::packages:
  'AsciiDoc':
    source: 'SublimeText/AsciiDoc'
  'Markdown-Preview':
    source: 'revolunet/sublimetext-markdown-preview'
  'PrettyJson':
    source: 'dzhibas/SublimePrettyJson'
  'Puppet':
    source: 'russCloak/SublimePuppet'

Tips: Hiera Structure

For example:

repo/config/hiera.yaml
---
:merge_behavior: deeper
:backends:
  - yaml
:yaml:
  :datadir: "%{::boxen_home}/repo/hiera"
:hierarchy:
  - "users/%{::github_login}/nodes/%{::hostname}"
  - "users/%{::github_login}/nodes/common"
  - "users/%{::github_login}"
  - "projects/%{::boxen_project_01}"
  - "projects/%{::boxen_project_02}"
  - "projects/%{::boxen_project_03}"
  - "projects/%{::boxen_project_04}"
  - "projects/%{::boxen_project_05}"
  - "projects/%{::boxen_project_06}"
  - "projects/%{::boxen_project_07}"
  - "projects/%{::boxen_project_08}"
  - "projects/%{::boxen_project_09}"
  - "projects/%{::boxen_project_10}"
  - "projects/common"
  - "common"

Tips: Roles

More advanced, but it can be worth borrowing some techniques from Puppet Enterprise

modules/people/jdigger/applications.pp
class people::jdigger::applications ($system_roles = undef) {
  $_system_roles = hiera_array('people::jdigger::system_roles', [])
  $roles = $system_roles ? { undef => $_system_roles, default => $system_roles}

  include people::jdigger::applications::general

  if member($roles, 'work') {
    include 'people::jdigger::applications::work'
  }
  if member($roles, 'personal') {
    include 'people::jdigger::applications::personal'
  }
}
hiera/users/jdigger/nodes/imac.yaml
boxen::security::require_password: false
people::jdigger::system_roles:
  - personal
modules/people/jdigger/applications/personal.pp
class people::jdigger::applications::personal {
  include 'calibre'
  include 'steam'
}

Troubleshooting

Summary

Resources

Attribution

Thank You!

Questions?

/

#