Securely Bootstrapping Munki Using Chef

In a previous article, I demonstrated a method of bootstrapping a new OS X client using Puppet’s client SSL certificates to secure Munki.

Continuing the topic of testing out Chef, I wanted to get similar behavior from a Chef setup that I can from a Puppet installation. The primary issue here is that Chef, unlike Puppet, doesn’t use built in client certificates – so we have to make them. I’ve previously written about setting up Chef with SSL client certificates, and setting up a Munki docker container to use Chef certificates.

The goal here is to be able to deploy a new computer with Chef and Munki preinstalled via DeployStudio (which runs over HTTPS), and then bootstrap Munki using SSL client certificates – meaning every part of the network deployment process is over a secure channel.

Strap in, because this one’s going to be complicated.

Process Overview

OS X Setup:

1. Follow the previously-blogged-about PKI process to get an SSL certificate on the Munki server, and on the OS X client.
2. Install OS X.
3. Install OS setup packages (admin account, skip registration, bypass setup assistant).
4. Add Chef & Munki servers to /etc/hosts if not in DNS.
5. Add Chef client.
6. Add Chef setup – client.rb and validation.pem files to /etc/chef/.
7. Add Munki & Munki configuration profile (using SSL client certificates).
8. Add Outset.
9. Add Chef first run script.
10. Add Chef trigger launchdaemon.

On First Boot:

1. Set the HostName, LocalHostName, and ComputerName.
2. Perform the initial Chef-client run using recipe [x509::munki2_client] to generate the CSR.
3. LaunchDaemon that waits for the existence of /etc/ssl/munki2.sacredsf.org.csr triggers:
   1. It will keep running the [x509::munki2_client] recipe while the CSR exists.
   2. The CSR will only be deleted when the CSR is signed on the Chef server.
   3. When the recipe succeeds and the CSR is removed and the .crt file is created, run the [munkiSSL::munki] recipe to copy the certificates into /Library/Managed Installs/certs/.
   4. Touch the Munki bootstrap file.
4. With the certificates in place, Munki ManagedInstalls profile installed, and the bootstrap trigger file present, Munki can now successfully bootstrap.

The Detailed Process:

Preparing the Deployment:

For my deployments, I like using Greg Neagle’s CreateOSXInstallPkg (henceforth referred to by acronym “COSXIP”) for generating OS X installer packages. Rather than crafting a specific image to be restored using DeployStudio, a package can be used to both install a new OS as well as upgrade-in-place over an existing OS.

One of the perks of using COSXIP is being able to load up additional packages that are installed at the same time as the OS, in the OS X Installer environment.

As mentioned above, we’re going to use a number of specific packages. Here’s what the COSXIP plist looks like:

	<?xml version="1.0" encoding="UTF-8"?>
	<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"&gt;
	<plist version="1.0">
	<dict>
	<key>Source</key>
	<string>/Applications/Install OS X Yosemite.app</string>
	<key>Output</key>
	<string>InstallYosemiteChefMunki.pkg</string>
	<key>Packages</key>
	<array>
	<string>AddChefToHostsDist.pkg</string>
	<string>AddMunkiToHostsDist.pkg</string>
	<string>chef-12.1.0-1.Dist.pkg</string>
	<string>ChefSetupDist.pkg</string>
	<string>ClearRegistrationSignedDist.pkg</string>
	<string>create_admin-fl-SignedDist-1.9.pkg</string>
	<string>ManagedInstalls-10.10-SSL-2.5.Dist.pkg</string>
	<string>Profile-SetupAssistant-10.10.2Dist.pkg</string>
	<string>munkitools-2.2.0.2399.pkg</string>
	<string>OutsetDist.pkg</string>
	<string>Outset-ChefClientDist.pkg</string>
	<string>XCodeCLITools.pkg</string>
	<string>ChefCSRTriggerDist.pkg</string>
	</array>
	<key>Identifier</key>
	<string>org.sacredsf.installosx.yosemite.pkg</string>
	</dict>
	</plist>

view raw

SecurelyBootstrappingMunkiChef_COSXIP.plist

hosted with ❤ by GitHub

Note that I’ve added “Dist” to the names of them. Due to a Yosemite requirement that all included packages be distribution packages, I have forcefully converted each package to a distribution using productbuild as described in the above link, and added “Dist” to the end to distinguish them.

The Packages:

AddChefToHosts and AddMunkiToHosts are payload-free packages that just add the IP addresses for my Chef and Munki2 server to /etc/hosts, since this is just in testing and those services don’t yet exist in DNS. The scripts look like this:

#!/bin/sh
echo "10.0.0.1 chef.sacredsf.org" >> "$3/private/etc/hosts"

chef-12.1.0-1.Dist is a specially repackaged-version of the Chef client. You can find the recipe for this in my AutoPkg repo.

The reason I did this is because the Chef-client’s postinstall script assumes that the target volume is a live booted OS X install – which is not true of the OS X install environment. The OS X install environment doesn’t have all OS X features, and the Chef client postinstall script will fail to do certain things like run basename and uname, and the symlinks will not work properly as they are executed in the OS install environment. My AutoPkg recipe addresses these issues and repackages the Chef client in a manner that is more compatible with the OS X install environment.

ChefSetup installs the client.rb and validation.pem files into /etc/chef/. The client.rb file looks like this:

	log_location STDOUT
	chef_server_url "https://chef.sacredsf.org:443/organizations/ssh"
	validation_client_name "ssh-validator"
	# Using default node name (fqdn)
	trusted_certs_dir "/etc/chef/trusted_certs"

view raw

SecurelyBootstrappingMunkiChef_Client.rb

hosted with ❤ by GitHub

The validation.pem file is the private key of the organization. See this blog post for details.

ClearRegistration, CreateAdmin, Profile-SetupAssistant are packages that bypass the OS X first-time boot setup process, by skipping the device registration, creating a local Admin account, and then skipping the iCloud Setup Assistant on first login. This allows me to boot straight to the Login Window and then login straight to the Desktop with no interruption.

ManagedInstalls-10.10-SSL installs the .mobileconfig profile that configures Munki. It enforces the settings that were accomplished using defaults in a previous blog post.

munkitools-2.2.0-2399 should be obvious.

Outset is the distribution package of Joseph Chilcote’s Outset, a fantastic tool for easily running scripts and packages at boot time and login time (which is easier than writing a new launch agent or launch daemon every time).

Outset-ChefClient installs the initial Chef setup script into /usr/local/outset/firstboot-scripts/. This initial Chef setup script looks like this:

	#!/bin/bash
	# Stolen from PSU:
	# https://wikispaces.psu.edu/display/clcmaclinuxwikipublic/First+Boot+Script
	echo "Starting run: `date`" >> /var/log/chef_outset.log
	echo "Waiting for network access" >> /var/log/chef_outset.log
	/usr/sbin/scutil -w State:/Network/Global/DNS -t 180
	sleep 5
	# Get the serial number
	serial=`system_profiler SPHardwareDataType | awk '/Serial/ {print $4}'`
	# If this is a VM in VMWare, Parallels, or Virtual Box, it might have weird serial numbers that Puppet doesn't like, so change it to something static
	if [[ `system_profiler SPHardwareDataType | grep VMware` || `system_profiler SPHardwareDataType | grep VirtualBox` || `system_profiler SPEthernetDataType | grep "/0x1ab8/"` ]]; then
	# Remove any silly + or / symbols
	serial="${serial//[+\/]}"
	fi
	/usr/sbin/scutil –set HostName "$serial.sacredsf.org"
	/usr/sbin/scutil –set LocalHostName "$serial-sacredsf-org"
	/usr/sbin/scutil –set ComputerName "$serial.sacredsf.org"
	echo "Hostname: `/usr/sbin/scutil –get HostName`" >> /var/log/chef_outset.log
	echo "LocalHostname: `/usr/sbin/scutil –get LocalHostName`" >> /var/log/chef_outset.log
	echo "ComputerName: `/usr/sbin/scutil –get ComputerName`" >> /var/log/chef_outset.log
	echo "Starting chef-client…" >> /var/log/chef_outset.log
	/usr/bin/chef-client –force-logger -L /var/log/chef_outset.log -l debug –once –runlist "recipe[x509::munki2_client]"
	echo "Finished chef-client." >> /var/log/chef_outset.log

view raw

SecurelyBootstrappingMunkiChef_ChefClientOutset

hosted with ❤ by GitHub

The script sets the hostname to the serial number (which I’m just using in my test environment so I can boot multiple VMs without having all of them be named “Mac.local”), and then runs the Chef client to trigger the generation of the CSR.

You can find the project for this in my GitHub repo.

XCodeCLITools installs the Xcode Command Line tools from the Developer site. This isn’t strictly necessary, but if you run Chef-client manually it will prompt you to install them, so preinstalling saves me some testing time.

ChefCSRTrigger installs the Launch Daemon that watches the path /etc/ssl/munki2.sacredsf.org.csr. So long as that path exists, this Launch Daemon will continue to trigger. The CSR is generated by the first run of the Outset Chef script, and this will keep making a request until the CA signs the CSR. The Launch Daemon looks like this:

	<?xml version="1.0" encoding="UTF-8"?>
	<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"&gt;
	<plist version="1.0">
	<dict>
	<key>Label</key>
	<string>org.sacredsf.chef.csrtrigger</string>
	<key>ProgramArguments</key>
	<array>
	<string>/usr/local/chef_munki_cert.sh</string>
	</array>
	<key>RunAtLoad</key>
	<false/>
	<key>KeepAlive</key>
	<dict>
	<key>PathState</key>
	<dict>
	<key>/etc/ssl/munki2.sacredsf.org.csr</key>
	<true/>
	</dict>
	</dict>
	<key>StandardOutPath</key>
	<string>/var/log/chef_csrtrigger.log</string>
	<key>StandardErrorPath</key>
	<string>/var/log/chef_csrtrigger.log</string>
	</dict>
	</plist>

view raw

SecurelyBootstrappingMunkiChef_CSRTrigger.plist

hosted with ❤ by GitHub

It runs this script:

	#!/bin/bash
	/usr/bin/chef-client –once –run-lock-timeout 120 –runlist "recipe[x509::munki2_client]"
	sleep 5
	if [[ -f /etc/ssl/munki2.sacredsf.org.crt ]]; then
	while [ ! -f /Library/Managed\ Installs/certs/clientcert.pem ]
	do
	/usr/bin/chef-client –once –run-lock-timeout 120 –runlist "recipe[munkiSSL::munki]"
	done
	touch /Users/Shared/.com.googlecode.munki.checkandinstallatstartup
	fi

view raw

SecurelyBootstrappingMunkiChef_MunkiCert.sh

hosted with ❤ by GitHub

Once the CSR is found, the script will attempt to run the same recipe again. If the recipe succeeds, the CSR will disappear and instead, /etc/ssl/munki2.sacredsf.org.crt will appear. If this file exists after the Chef-client run, the script will proceed to try and run the [munkiSSL::munki] recipe until it has successfully copied over the cert into /Library/Managed Installs/certs/clientcert.pem (which should theoretically only take one run). Then, it will create the Munki bootstrap file.

You can find this project in my GitHub repo.

With all of these packages, you can build your OS X installer to use in DeployStudio:
sudo ./createOSXinstallPkg --plist=InstallYosemite-ChefMunki.plist

Deployment:

When a computer is NetBooted into DeployStudio, and the OS is installed (along with all the packages above), that’s when the fun stuff happens.

1. On first boot, Outset will execute the run_chef.sh script (installed by the Outset-ChefClient package). This script will wait for network access, and then use scutil to set the HostName, LocalHostName, and ComputerName to the serial number. Then, it will run execute the first Chef client run with the [x509::munki2_client] recipe, which generates a private key and submits a CSR to the Chef server.

2. The creation of the CSR file at /etc/ssl/munki2.sacredsf.org.csr triggers the execution of the org.sacredsf.chef.csrtrigger LaunchDaemon (installed by the ChefCSRTrigger package), which will continually run the chef_munki_cert.sh script while that CSR file is present.

3. On the Chef server/workstation, the CSR needs to be signed (this is assuming the ChefCA is set up according to previous blog posts):
   chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA

4. When the CSR is signed, the LaunchDaemon that is spinning in circles around the CSR file will finally have a successful chef-client run. The successful run will delete the csr file and create the signed certificate file at /etc/ssl/munki2.sacredsf.org.crt.

5. Once this file exists, the script will then trigger the [munkiSSL::munki] recipe, which copies the certificates and private keys from /etc/ssl/ into /Library/Managed Installs/certs/ with the appropriate names.

6. Finally, the Munki bootstrap file is created at /Users/Shared/.com.googlecode.munki.checkandinstallatstartup.

7. The appearance of the Bootstrap file will cause Munki to execute immediately (as we’re still at the Login Window at this point). Munki will read the preferences from the ManagedInstalls profile settings, which tells it to use the certificates in /Library/Managed Installs/certs/ to check the server https://munki2.sacredsf.org/repo for updates.

8. If the certificates are valid, Munki will proceed with a normal bootstrap run, except through a secure SSL connection that uses its client certificates to communicate with the Munki server, which has been configured to require these certificates (see the previous blog posts).

Conclusion

It’s now possible to securely bootstrap a new OS X machine using Chef to set up SSL client certificates to use with Munki. The best part is that it doesn’t require hands-on attention on the OS X client. The downside is that, at this point, it does require hands-on attention on the Chef server, where the CA is.

There are some possible easy fixes for that, though. The easiest solution would be to run a cronjob on the Chef server that automatically signs all CSRs every X amount of time, which would eliminate any need for manual intervention on the Chef CA. That’s not a desirable method, though, because that’s essentially letting any client who runs the right recipe get a free SSL certificate to the Munki repo. There’s no verification that the client is one we want to allow.

Another possibility to use a more industrial-strength internal CA not managed by Chef, which can have its own policies and methods for signing certificates. This is more common in enterprise environments where they tend to have their own root CAs and Intermediary CAs for internal-only services. More commercial offerings of this sort of thing probably have better methods for determining which CSRs get signed and which don’t.

The chef-ssl client can also be used to generate CSRs for third-party external CAs, but you probably wouldn’t want to sign individual clients with an external CA.

At least we can bootstrap a large batch of machines at once. With 30 machines running, they’ll all submit CSRs and sit there waiting until they get signed. In one command on the CA server, you can sign all 30 CSRs and they’ll automatically proceed to the next step, which is to get the certs and then bootstrap Munki. So we’re at mostly unattended install. But hey, as a proof of concept, it works!