Local-Only Manifests in Munki

A while back, there was a discussion on Munki-Dev floating the idea of local-only manifests. After some long discussion, the final Pull Request was created and merged.

The idea behind local-only manifests is simple: if you specify a LocalOnlyManifest key in the preferences, Munki will look for that manifest name in /Library/Managed Installs/manifests. If it finds it, it’ll look for any managed_installs and managed_uninstalls specified inside, and concatenate those with whatever it gets from the Munki server. It’s an extra place to specify managed installs and uninstalls that is unique to the client.

Essentially, what it does is move the unique-client logic from the server to the client. As you scale upwards in client numbers, having huge numbers of unique server-side manifests induces significant overhead – potentially 10,000+ unique manifests in your Munki server’s manifests directory gets unwieldy. With the uniqueness moved client-side, the server only has to provide the common manifests.

There’s a lot of neat things you can do with this idea, so let’s explore some of them!

Hang Out With The Locals

While the basic idea of the local-only manifest is simple, the implementation has some fun details you can take advantage of.

Local-only manifests do not have any catalogs of their own. Instead, they inherit from whatever catalog is provided by the manifest given from the ClientIdentifier key. Thus, if your main manifest uses the catalog “release”, any items specified in the local-only manifest must also be in the “release” catalog (or they will simply be treated like adding any item to a manifest when it is not in a catalog – which is to say that you will receive warnings).

Local-only manifests also don’t have their own conditional items. This is where interaction with third-party tools really begins to shine, but we’ll explore that later.

Because this is a unique manifest, you get the benefits that “real” manifests get. You can specify items to be installed here that are not provided as optional items in the server-side manifest (as long as they’re in the catalog). You can still get the server’s provided list of optional installs, and use the local-only manifest to determine what items become managed installs or removals.

This doesn’t absolve the Munki admin of taking care, though. It’s still possible for an item to be specified as a managed install in one manifest and a managed uninstall in another manifest – and therefore trigger a collision. Local-only manifests are just as vulnerable to that as server-side manifests, and it’s easy for a client to contravene the server-side manifest and result in undefined (or undesireable) behavior.

It’s my recommendation, therefore, that you split the purposes and logic behind the server-side and local-only manifests into separate functions – optional vs. mandatory.

One Manifest To Rule Them All

Because of the slightly limited nature of local-only manifests, it’s important to think of them as addenda to server-side manifests. The way to mentally separate these functions is to also separate “mine” vs. “yours” – the things I, the Munki admin, want your machine to have vs. the things you, the client, want your machine to have (or not have).

The easiest way to accomplish this is to completely remove managed_installs and managed_uninstalls from your server-side manifest. The server-side manifest thus becomes the self-service list and gatekeeper to all optional software. The Munki admins determine what software is available because they control the optional installs list as well as the catalogs, but the clients now have essentially free customizability without needing any ability to modify the servers.

Because the unique aspects of clients are now done client-side and not server-side, this allows an external management mechanism, like Chef or Puppet, to control what Munki manages on a client, without needing the ability to make changes to the repo. If your repo is in source control (and it should be!), this means that the only commits to the repo’s manifests are done by the Munki admins, and will only involve changes that generally affect the whole fleet.

Whence Does This Mystical Manifest Come From?

The local-only manifest moves the work from maintaining the manifest relationships on the server to maintaining them on the client. This is really only beneficial if you already have a mechanism in place to manage these files – such as a config management tool (Chef, Puppet, etc.).

Facebook CPE handles this with our cpe_munki cookbook for Chef. In addition to managing the installation and configuration of Munki, we also create a local-only manifest on disk and tell clients to use it. Manifests are just plists, and plists are just structured-data representations of dictionaries/hashes.

Nearly every programming language offers a mechanism for interacting with dictionaries/hashes in relatively easy ways, and Ruby (in both Chef and Puppet) allows for simple abstractions here.

Abstracting Local Manifests Into Simple Variables

I’m going to use pseudo-Ruby via Chef as the base for this, but the same principles will apply to any scripting language or tool.

The Process in pseudocode:

The point of the pseudocode above is to show how simple it is to abstract out what amounts to a complex process – deciding what software is installed or removed on a machine – and reduce it to simply two arrays.

To add something to be installed on your client, you add to the local managed installs variable. Same for removals and its equivalent variable.

What you now have here is a mechanism by which you can use any kind of condition or trigger as a result of your config management engine to determine what gets installed on individual clients.

Use Some Conditioning, It Makes It All Smooth

Veteran Munki admins are very familiar with conditional items. Conditions can be used to place items in appropriate slots – managed installs/uninstalls, optionals, etc. They’re an extremely powerful aspect of manifests, and allows for amazing and complex logic and customization. You can also provide your own conditions using admin-provided conditionals, which essentially allow you to script any logic you want for this purpose.

Conditions in Munki are critical to success, but NSPredicates can be difficult and unintuitive. Admin-provided conditionals are a convenient way to get around complex NSPredicate logic by scripting what you want, but they require multiple steps:

  1. You have to write the scripting logic,
  2. You have to deploy the conditional scripts to the clients
  3. You still have to write the predicates into the manifest.

They’re powerful but require some work to utilize.

In the context of a local-only manifest, though, all of the logic for determining what goes in is determined entirely your management system. So there’s technically no client-side evaluation of predicates happening, because that logic is handled by the management engine whenever it runs. This unifies your logic into a single codebase which makes maintaining it easy, with less moving parts overall.

Some Code Examples

This is all implemented in Chef via IT CPE’s cpe_munki implementation, but here I’m going to give some examples of how to take this abstraction and use it.

In Chef, the local-only managed_installs is expressed as a node attribute, which is essentially a persistent variable throughout an entire Chef run. This array represents an array of strings – a list of all the item names from Munki that will be added to managed installs.

Thus, adding items in Chef is easy as pie:

node.default['cpe_munki']['local']['managed_installs'] << 'GoogleChrome'

Same goes for managed uninstalls:

node.default['cpe_munki']['local']['managed_uninstalls'] << 'MacKeeper'

Additionally, we specify in the Munki preferences that we have a local-only manifest called “extra_packages”:

 'DaysBetweenNotifications' => 90,
 'InstallAppleSoftwareUpdates' => true,
 'LocalOnlyManifest' => 'extra_packages',
 'UnattendedAppleUpdates' => true,
 }.each do |k, v|
   node.default['cpe_munki']['preferences'][k] = v

After a Chef run, you’ll see the file in /Library/Managed Installs/manifests:

$ ls -1 /Library/Managed\ Installs/manifests

If you look inside that file, you’ll see a plist with your managed installs and removals:


When managedsoftwareupdate runs, it will concatenate the server-side manifest with the local-manifest, as described above. The sample plist above will ensure that six items are always going to be installed by Munki on my machine, and that “Tableau8” will always attempt to uninstall if needed.

With a setup like this, anyone who can submit code to the Chef repo can easily configure their machine for whatever settings they want, and thus users have individual control over their own machines without needing the ability to access any of the server manifests.

Even If You Don’t Have Config Management

You can still benefit from local-only manifests without needing config management. Manifests, including local ones, are just plists, and there are lots of ways to manipulate plists already available.

You could also add items to your local manifest using defaults:

$ sudo defaults write /Library/Managed\ Installs/manifests/extra_packages managed_installs -array-add "GoogleChrome"

Note the issue mentioned above, though, which is that it’s trivial for someone to add an item name that doesn’t exist in the catalog. Should that happen, the Munki client would generate warnings to your reporting engine. The benefits of using an external config management is the ability to lint or filter out non-existent items and thus prevent such warnings.


Ultimately, the benefits here are obvious. Clients have the ability to configure themselves without needing any access to the Munki repo. In addition, your users and customers don’t even need to have an understanding of manifests or how they work in order to get results. The entire interaction they’ll have with Munki will be understanding that items added to managed_installs get installed, and items added to managed_uninstalls get removed.

Stay tuned for a follow-up blog post about how this fits into Facebook’s overall managed Munki strategy, and how source control plays an important role in this process.


Generating PBKDF2 Password Hashes In Python, Not Ruby

Chef offers a great many useful features, including the ability to manage and create user accounts. The password for a local user account can be specified either in clear text or as a password hash.

According to the documentation linked above, generating an appropriate password hash for 10.8+ requires the use of a specific Ruby function:


However, when trying to generate such a hash using this tool on 10.10.5, I discovered a problem:

irb(main):026:0> OpenSSL::PKCS5::pbkdf2_hmac(
irb(main):027:1* password,
irb(main):028:1* salt,
irb(main):029:1* iterations,
irb(main):030:1* 128,
irb(main):031:1* OpenSSL::Digest::SHA512.new
irb(main):032:1> )
NotImplementedError: pbkdf2_hmac() function is unimplemented on this machine
from (irb):26:in `pbkdf2_hmac'
from (irb):26
from /usr/bin/irb:12:in `<main>'

Well, that's not very nice.

The issue is that the version of OpenSSL on OS X for the last several years is still 0.9.8zg. That version simply doesn't have an implementation of the pbkdf2_hmac() that Ruby wants to use. However, Python does, thanks to hashlib.

To recreate the same process in Python that the Chef documentation recommends for generating a 10.8+ password hash, use the following steps:

import hashlib
import binascii
import os

password = b'password'
salt = os.urandom(32)
chef_salt = binascii.hexlify(salt)
iterations = 25000

hex = hashlib.pbkdf2_hmac('sha512', password, salt, 25000, 128)
chef_password_hash = binascii.hexlify(hex)

Let's break down what happened there. First, we set the password to our password string. In Python 2, the b before a string doesn't really do anything.

The salt is a random 32-bit string. In Python, this comes out in binary form:

>>> salt = os.urandom(32)
>>> salt

Chef, however, requires this in a hexadecimal form:

>>> binascii.hexlify(salt)

We use 25,000 iterations as a nice arbitrary number, but you should use anything above 10,000 to be at least minimally safe. Of course this is a local user account on a service machine in my context, so I’m not entirely worried about its security as much.

Once we have all the variables, we can use the actual pbkdf2_hmac() function. In the example above, we’re using the SHA-512 digest, with a derived key length of 128 as the Chef documentation suggests. Once again, the result of that command is binary data, and Chef requires a hexadecimal representation, so we turn to our trusty friend binascii.hexlify again.

This allows us to create the Chef user block we need:

user 'myuser' do
gid 'staff'
home '/Users/myuser'
shell '/bin/bash'
password 'e6a8a452c0a9edb7f80703657b91fae74191d3b83982687ca00b83741ad775410178542ffc176abe6db9dc46053bc7ed36c91c1f43f82ba1dedc12de929f81cca868e223a25f3f16728e9f92c02e4421e9f73d73edb5e23e5d0cf1784243e8c79307ee5e61b411c9f116c450af8112e519fa15cfb50f5e7a8c1e6a78fb7cbc0e'
salt 'eb30e9c1946f086b4cd84679c1ee81235edea080b28b1ce4d39341794fad1ccd'
iterations 25_000
supports manage_home: true
action :create

I’m told this same technique can also generate password hashes to be used with Puppet as well, although I haven’t tested it personally.

Securely Bootstrapping Munki Using Chef

In a previous article, I demonstrated a method of bootstrapping a new OS X client using Puppet’s client SSL certificates to secure Munki.

Continuing the topic of testing out Chef, I wanted to get similar behavior from a Chef setup that I can from a Puppet installation. The primary issue here is that Chef, unlike Puppet, doesn’t use built in client certificates – so we have to make them. I’ve previously written about setting up Chef with SSL client certificates, and setting up a Munki docker container to use Chef certificates.

The goal here is to be able to deploy a new computer with Chef and Munki preinstalled via DeployStudio (which runs over HTTPS), and then bootstrap Munki using SSL client certificates – meaning every part of the network deployment process is over a secure channel.

Strap in, because this one’s going to be complicated.

Process Overview

OS X Setup:

  1. Follow the previously-blogged-about PKI process to get an SSL certificate on the Munki server, and on the OS X client.
  2. Install OS X.
  3. Install OS setup packages (admin account, skip registration, bypass setup assistant).
  4. Add Chef & Munki servers to /etc/hosts if not in DNS.
  5. Add Chef client.
  6. Add Chef setup – client.rb and validation.pem files to /etc/chef/.
  7. Add Munki & Munki configuration profile (using SSL client certificates).
  8. Add Outset.
  9. Add Chef first run script.
  10. Add Chef trigger launchdaemon.

On First Boot:

  1. Set the HostName, LocalHostName, and ComputerName.
  2. Perform the initial Chef-client run using recipe [x509::munki2_client] to generate the CSR.
  3. LaunchDaemon that waits for the existence of /etc/ssl/munki2.sacredsf.org.csr triggers:
    1. It will keep running the [x509::munki2_client] recipe while the CSR exists.
    2. The CSR will only be deleted when the CSR is signed on the Chef server.
    3. When the recipe succeeds and the CSR is removed and the .crt file is created, run the [munkiSSL::munki] recipe to copy the certificates into /Library/Managed Installs/certs/.
    4. Touch the Munki bootstrap file.
  4. With the certificates in place, Munki ManagedInstalls profile installed, and the bootstrap trigger file present, Munki can now successfully bootstrap.

The Detailed Process:

Preparing the Deployment:

For my deployments, I like using Greg Neagle’s CreateOSXInstallPkg (henceforth referred to by acronym “COSXIP”) for generating OS X installer packages. Rather than crafting a specific image to be restored using DeployStudio, a package can be used to both install a new OS as well as upgrade-in-place over an existing OS.

One of the perks of using COSXIP is being able to load up additional packages that are installed at the same time as the OS, in the OS X Installer environment.

As mentioned above, we’re going to use a number of specific packages. Here’s what the COSXIP plist looks like:

Note that I’ve added “Dist” to the names of them. Due to a Yosemite requirement that all included packages be distribution packages, I have forcefully converted each package to a distribution using productbuild as described in the above link, and added “Dist” to the end to distinguish them.

The Packages:

AddChefToHosts and AddMunkiToHosts are payload-free packages that just add the IP addresses for my Chef and Munki2 server to /etc/hosts, since this is just in testing and those services don’t yet exist in DNS. The scripts look like this:

echo " chef.sacredsf.org" >> "$3/private/etc/hosts"

chef-12.1.0-1.Dist is a specially repackaged-version of the Chef client. You can find the recipe for this in my AutoPkg repo.

The reason I did this is because the Chef-client’s postinstall script assumes that the target volume is a live booted OS X install – which is not true of the OS X install environment. The OS X install environment doesn’t have all OS X features, and the Chef client postinstall script will fail to do certain things like run basename and uname, and the symlinks will not work properly as they are executed in the OS install environment. My AutoPkg recipe addresses these issues and repackages the Chef client in a manner that is more compatible with the OS X install environment.

ChefSetup installs the client.rb and validation.pem files into /etc/chef/. The client.rb file looks like this:

The validation.pem file is the private key of the organization. See this blog post for details.

ClearRegistration, CreateAdmin, Profile-SetupAssistant are packages that bypass the OS X first-time boot setup process, by skipping the device registration, creating a local Admin account, and then skipping the iCloud Setup Assistant on first login. This allows me to boot straight to the Login Window and then login straight to the Desktop with no interruption.

ManagedInstalls-10.10-SSL installs the .mobileconfig profile that configures Munki. It enforces the settings that were accomplished using defaults in a previous blog post.

munkitools-2.2.0-2399 should be obvious.

Outset is the distribution package of Joseph Chilcote’s Outset, a fantastic tool for easily running scripts and packages at boot time and login time (which is easier than writing a new launch agent or launch daemon every time).

Outset-ChefClient installs the initial Chef setup script into /usr/local/outset/firstboot-scripts/. This initial Chef setup script looks like this:

The script sets the hostname to the serial number (which I’m just using in my test environment so I can boot multiple VMs without having all of them be named “Mac.local”), and then runs the Chef client to trigger the generation of the CSR.

You can find the project for this in my GitHub repo.

XCodeCLITools installs the Xcode Command Line tools from the Developer site. This isn’t strictly necessary, but if you run Chef-client manually it will prompt you to install them, so preinstalling saves me some testing time.

ChefCSRTrigger installs the Launch Daemon that watches the path /etc/ssl/munki2.sacredsf.org.csr. So long as that path exists, this Launch Daemon will continue to trigger. The CSR is generated by the first run of the Outset Chef script, and this will keep making a request until the CA signs the CSR. The Launch Daemon looks like this:

It runs this script:

Once the CSR is found, the script will attempt to run the same recipe again. If the recipe succeeds, the CSR will disappear and instead, /etc/ssl/munki2.sacredsf.org.crt will appear. If this file exists after the Chef-client run, the script will proceed to try and run the [munkiSSL::munki] recipe until it has successfully copied over the cert into /Library/Managed Installs/certs/clientcert.pem (which should theoretically only take one run). Then, it will create the Munki bootstrap file.

You can find this project in my GitHub repo.

With all of these packages, you can build your OS X installer to use in DeployStudio:
sudo ./createOSXinstallPkg --plist=InstallYosemite-ChefMunki.plist


When a computer is NetBooted into DeployStudio, and the OS is installed (along with all the packages above), that’s when the fun stuff happens.

  1. On first boot, Outset will execute the run_chef.sh script (installed by the Outset-ChefClient package). This script will wait for network access, and then use scutil to set the HostName, LocalHostName, and ComputerName to the serial number. Then, it will run execute the first Chef client run with the [x509::munki2_client] recipe, which generates a private key and submits a CSR to the Chef server.

  2. The creation of the CSR file at /etc/ssl/munki2.sacredsf.org.csr triggers the execution of the org.sacredsf.chef.csrtrigger LaunchDaemon (installed by the ChefCSRTrigger package), which will continually run the chef_munki_cert.sh script while that CSR file is present.

  3. On the Chef server/workstation, the CSR needs to be signed (this is assuming the ChefCA is set up according to previous blog posts):
    chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA

  4. When the CSR is signed, the LaunchDaemon that is spinning in circles around the CSR file will finally have a successful chef-client run. The successful run will delete the csr file and create the signed certificate file at /etc/ssl/munki2.sacredsf.org.crt.

  5. Once this file exists, the script will then trigger the [munkiSSL::munki] recipe, which copies the certificates and private keys from /etc/ssl/ into /Library/Managed Installs/certs/ with the appropriate names.

  6. Finally, the Munki bootstrap file is created at /Users/Shared/.com.googlecode.munki.checkandinstallatstartup.

  7. The appearance of the Bootstrap file will cause Munki to execute immediately (as we’re still at the Login Window at this point). Munki will read the preferences from the ManagedInstalls profile settings, which tells it to use the certificates in /Library/Managed Installs/certs/ to check the server https://munki2.sacredsf.org/repo for updates.

  8. If the certificates are valid, Munki will proceed with a normal bootstrap run, except through a secure SSL connection that uses its client certificates to communicate with the Munki server, which has been configured to require these certificates (see the previous blog posts).


It’s now possible to securely bootstrap a new OS X machine using Chef to set up SSL client certificates to use with Munki. The best part is that it doesn’t require hands-on attention on the OS X client. The downside is that, at this point, it does require hands-on attention on the Chef server, where the CA is.

There are some possible easy fixes for that, though. The easiest solution would be to run a cronjob on the Chef server that automatically signs all CSRs every X amount of time, which would eliminate any need for manual intervention on the Chef CA. That’s not a desirable method, though, because that’s essentially letting any client who runs the right recipe get a free SSL certificate to the Munki repo. There’s no verification that the client is one we want to allow.

Another possibility to use a more industrial-strength internal CA not managed by Chef, which can have its own policies and methods for signing certificates. This is more common in enterprise environments where they tend to have their own root CAs and Intermediary CAs for internal-only services. More commercial offerings of this sort of thing probably have better methods for determining which CSRs get signed and which don’t.

The chef-ssl client can also be used to generate CSRs for third-party external CAs, but you probably wouldn’t want to sign individual clients with an external CA.

At least we can bootstrap a large batch of machines at once. With 30 machines running, they’ll all submit CSRs and sit there waiting until they get signed. In one command on the CA server, you can sign all 30 CSRs and they’ll automatically proceed to the next step, which is to get the certs and then bootstrap Munki. So we’re at mostly unattended install. But hey, as a proof of concept, it works!

Running Munki with Chef SSL Client Certificates in Docker

Previously, I wrote about building a Docker container for Munki with Chef installed. Having built that container, it’s now time to put it to use.

Assuming you’ve got a working Chef server set up, we can run our Munki-with-Chef container and register it.

Preparing the Server:

First, we need to set up the CA we’re going to use with Munki, via Chef. This will also assume you have a chef repo in ~/chef-repo/ that is set up according to the Chef documentation.

I’ve described the general process in a previous blog post here, but I’ve changed enough of it that I’m going to repeat a lot of it here.

  1. On the server/workstation, download the cookbook:
    knife cookbook site install x509
  2. Now delete it and clone my version:
    git clone --branch development https://github.com/nmcspadden/chef-cookbook-ssl.git x509
  3. Clone the MunkiSSL cookbook from Github:
    git clone https://github.com/nmcspadden/chef-cookbook-munkiSSL munkiSSL
  4. Upload all the cookbooks to the server:
    knife cookbook upload -a
  5. Create the ‘certificates’ data bag:
    knife data bag create certificates
  6. Create the CA (I’m storing it in my home directory for this example):
    chef-ssl makeca --dn '/CN=ChefCA' --ca-path /home/nmcspadden/chefCA
    Pick any passphrase you want.

Running the Container:

In this blog post, I’m going to call this container “munki2” so as not to interfere with my existing Munki container.

Prepare a data-only container to keep our data in:
docker run -d --name munki2-data --entrypoint /bin/echo nmcspadden/munki-chef Data-only container for munki2

Run the container:
docker run -d --name munki2 --volumes-from munki2-data -p 443:443 -h munki2.domain.com nmcspadden/munki-chef

This run command sets the open port to 443, for SSL connections, and uses the munki2-data data container to access the repo. In addition, I’ve set the hostname here manually as well, using the -h option.

If you are just testing this out and don’t have your Chef server entered into your DNS, you can fix that inside the container using the --add-host option like so:
docker run -d --name munki2 --volumes-from munki2-data -p 443:443 -h munki2.domain.com --add-host chef.domain.com: nmcspadden/munki-chef

The first step after running the container is to check in with Chef:
docker exec munki2 /usr/bin/chef-client --force-logger --runlist "recipe[x509::munki2_server]"

Here, we’re using the [x509::munki2_server] recipe to generate a private key and send a CSR to the Chef CA.

On the Chef server or workstation, you’ll need to sign the CSR:
chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA

Back on the Docker host, run the x509::munki2_server recipe again to receive the signed certificate:
docker exec munki2 /usr/bin/chef-client --force-logger --runlist "recipe[x509::munki2_server]"

Screenshot 2015-03-03 14.41.25

You can verify the certificate’s existence on the Chef server / workstation:
knife search certificates "host:munki2.domain.com" -a dn

Now that the certificates are present, it’s time to add in the new Nginx config file to tell the webserver to use client certificates:
cat munki-repo-ssl.conf | docker exec -i munki2 sh -c 'cat &gt; /etc/nginx/sites-enabled/munki-repo.conf'

The Nginx configuration looks like this:

The three important file paths that must be correct are ssl_certificate, ssl_certificate_key, and ssl_client_certificate. If any of these paths are wrong or can’t be found, Nginx will not start and your Docker container will immediately halt.

For reference, the ssl_protocols and ssl_ciphers are configured for perfect forward secrecy.

Otherwise, the configuration for Nginx for the Munki repo remains the same as the non-SSL version – we’re serving the file path /munki_repo as https://munki2/repo/.

Now restart the container to reload the Nginx configuration:

docker stop munki2 && docker start munki2

We have a working Munki server on port 443, now it needs to be populated.

Configure the Clients to use Munki with SSL certificates:

If you are testing this out, you probably don’t have munki2 or Chef in your DNS entry. You’ll need to add them to your /etc/hosts file on the clients first: munki2 munki2.domain.com chef chef.domain.com

Detailed instructions on configuring Munki with SSL certificates can be found on the official wiki, but I’m going to recreate the steps here.

  1. On the client, run the [x509::munki2_client] recipe to generate a CSR:
    sudo chef-client --runlist "recipe[x509::munki2_client]"
  2. On the Chef Server or workstation, use chef-ssl to sign the CSR:
    chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA
  3. On the OS X client, run the recipe again to receive the signed certificate:
    sudo chef-client --runlist "recipe[x509::munki2_client]"
  4. With the client set up with its certificate, now it’s time to configure Munki. Run the [munkiSSL::munki] recipe:
    sudo chef-client --runlist "recipe[munkiSSL::munki]"
    This recipe copies the client certificates from /etc/ssl/ into /Library/Managed Installs/certs/ where Munki can use them.
  5. Change the ManagedInstalls.plist defaults:
    1. sudo defaults write /Library/Preferences/ManagedInstalls SoftwareRepoURL "https://munki2/repo"
    2. sudo defaults write /Library/Preferences/ManagedInstalls SoftwareRepoCACertificate "/Library/Managed Installs/certs/ca.pem"
    3. sudo defaults write /Library/Preferences/ManagedInstalls ClientCertificatePath "/Library/Managed Installs/certs/clientcert.pem"
    4. sudo defaults write /Library/Preferences/ManagedInstalls ClientKeyPath "/Library/Managed Installs/certs/clientkey.pem"
    5. sudo defaults write /Library/Preferences/ManagedInstalls UseClientCertificate -bool TRUE
  6. Finally, test out the client:
    sudo /usr/local/munki/managedsoftwareupdate -vvv --checkonly

Using Chef to Set Up SSL Client Certificates

Previously, I’ve written about setting up Chef Server 12, and then configuring it to use a trusted SSL certificate.

Now, we’re going to go another step, which is to use Chef to generate client certificates by creating a CA, and signing CSRs from clients.

This is based on the documentation for setting up a simple x509 PKI infrastructure with Chef, based on the x509 cookbook. That blog post is very helpful, but is missing some crucial steps, so I’m going to recreate it here and fill in some of the blanks.

Note: Despite my best efforts, I could not get this to work with the default Chef self-signed certificate. This only functioned when I switched to a real SSL certificate. See my previous blog post for a guide.

Set Up the Server:

Most of these instructions will require root or sudo. Also, it’s helpful to use the Ruby provided by Chef, so check which ruby to make sure you’re using the one in /opt/chefdk/embedded/bin/ruby
Let’s go through the process of setting up the x509 cookbook to create a CA for us:

  1. Install the chef-ssl-client gem:
    gem install chef-ssl-client
  2. Install the x509 cookbook:
    knife cookbook site install x509
  3. Replace the x509 cookbook with a fork I created to fix some gaps in the OS X implementation:
    rm -rf chef-repo/cookbooks/x509
    git clone --branch OSX https://github.com/nmcspadden/chef-cookbook-ssl.git x509
  4. Create the ‘certificates’ data bag:
    knife data bag create certificates
  5. Create a test recipe that will install the certificates on the client, located at chef-repo/cookbooks/x509/recipes/chef-ca.rb:
    include_recipe "x509::default"
    x509_certificate "chef.sacredsf.org" do
    	certificate "/etc/ssl/chef.sacredsf.org.cert"
    	cacertificate "/etc/ssl/chef_ca.cert"
    	key "/etc/ssl/chef.sacredsf.org.key" 
    	ca "ChefCA" 
    	type "server" 
    	bits 2048 
    	days 365 
  6. Now that the recipe is created, we need to upload the cookbooks to the server:
    knife cookbook upload -a
    That command uploads all cookbooks. If you have more than the x509 and vt-gpg cookbooks installed, you can be more precise:
    knife cookbook upload x509 vt-gpg
  7. Finally, create the actual CA. Substitute the ca-path with a valid one of your own:
    chef-ssl makeca --dn '/CN=ChefCA' --ca-path /home/nmcspadden/chefCA
    Enter a passphrase when prompted.

Test It:

Now, it’s time to test it on a Chef client. I tested this on a Mac OS 10.10.2 Yosemite VM, which I set up in the original article about OS X client configuration:
sudo chef-client --force-logger --runlist "recipe[x509::chef-ca]"

You should see output similar to this:

[2015-02-25T13:59:46-08:00] INFO: Processing x509_certificate[chef.sacredsf.org] action create (x509::chef-ca line 3)
[2015-02-25T13:59:46-08:00] INFO: Processing directory[/etc/ssl] action create (/var/chef/cache/cookbooks/x509/providers/certificate.rb line 11)
[2015-02-25T13:59:46-08:00] INFO: directory[/etc/ssl] created directory /etc/ssl
[2015-02-25T13:59:46-08:00] INFO: directory[/etc/ssl] mode changed to 755
[2015-02-25T13:59:46-08:00] INFO: Processing file[/etc/ssl/chef.sacredsf.org.cert] action create (/var/chef/cache/cookbooks/x509/providers/certificate.rb line 17)
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.cert] created file /etc/ssl/chef.sacredsf.org.cert
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.cert] updated file contents /etc/ssl/chef.sacredsf.org.cert
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.cert] owner changed to 0
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.cert] group changed to 0
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.cert] mode changed to 644
[2015-02-25T13:59:46-08:00] INFO: Processing file[/etc/ssl/chef.sacredsf.org.key] action create (/var/chef/cache/cookbooks/x509/providers/certificate.rb line 23)
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.key] created file /etc/ssl/chef.sacredsf.org.key
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.key] updated file contents /etc/ssl/chef.sacredsf.org.key
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.key] owner changed to 0
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.key] group changed to 0
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.key] mode changed to 600
[2015-02-25T13:59:46-08:00] INFO: Processing file[/etc/ssl/chef_ca.cert] action create (/var/chef/cache/cookbooks/x509/providers/certificate.rb line 30)
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef_ca.cert] created file /etc/ssl/chef_ca.cert
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef_ca.cert] updated file contents /etc/ssl/chef_ca.cert
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef_ca.cert] owner changed to 0
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef_ca.cert] group changed to 0
[2015-02-25T13:59:46-08:00] INFO: file[/etc/ssl/chef_ca.cert] mode changed to 644
[2015-02-25T13:59:46-08:00] INFO: Chef Run complete in 4.990408 seconds

Back on the Chef server / workstation, you can check the CSR in the node’s data. If you don’t know the name of the node, run this command to get a list:
knife node list
In my case, my node’s name is TestVM.local.

Check the CSR:
knife node show TestVM.local -a csr_outbox

      ca:   ChefCA
      csr:  -----BEGIN CERTIFICATE REQUEST-----
      date: 2015-02-25 13:59:46 -0800
      days: 365
      id:   0f68afcc64e13af040bdf9f6b53d4c701c3614d5c6fb1cb7df5751dbbb92d0aa
      type: server

Sign It:

Now we can sign the CSR on the server:
chef-ssl autosign --ca-name="ChefCA" --ca-path=/home/nmcspadden/chefCA
Enter the passphrase you used earlier to generate the CA.

This command will search for all outstanding CSRs and ask you to sign them (which is horribly inefficient and requires lots of manual effort for production use, but that’s a future topic for discussion).

Once you have signed all outstanding CSRs (in this example, only one) by entering “yes” when prompted, the run completes:
Saved OK
All CSRs processed.

Signed certificates are saved into the ‘certificates’ data bag:
knife search certificates "host:TestVM.local" -a dn

1 items found

  dn: /C=GB/ST=London/L=London/O=Example Ltd/OU=Certificate Automation/CN=chef.sacredsf.org/emailAddress=x509-auto@example.com

To search for all certificates:
knife search certificates "host:*" -a dn

Finally, return to the client and run chef-client again to complete the certificate generation:
sudo chef-client --force-logger --runlist "recipe[x509::chef-ca]"

[2015-02-25T15:02:11-08:00] INFO: Processing x509_certificate[chef.sacredsf.org] action create (x509::chef-ca line 3)
[2015-02-25T15:02:11-08:00] INFO: installing certificate chef.sacredsf.org (id 0f68afcc64e13af040bdf9f6b53d4c701c3614d5c6fb1cb7df5751dbbb92d0aa)
[2015-02-25T15:02:11-08:00] INFO: Processing directory[/etc/ssl] action create (/var/chef/cache/cookbooks/x509/providers/certificate.rb line 11)
[2015-02-25T15:02:11-08:00] INFO: Processing file[/etc/ssl/chef.sacredsf.org.cert] action create (/var/chef/cache/cookbooks/x509/providers/certificate.rb line 17)
[2015-02-25T15:02:11-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.cert] backed up to /var/chef/backup/etc/ssl/chef.sacredsf.org.cert.chef-20150225150211.867169
[2015-02-25T15:02:11-08:00] INFO: file[/etc/ssl/chef.sacredsf.org.cert] updated file contents /etc/ssl/chef.sacredsf.org.cert
[2015-02-25T15:02:11-08:00] INFO: Processing file[/etc/ssl/chef.sacredsf.org.key] action nothing (/var/chef/cache/cookbooks/x509/providers/certificate.rb line 23)
[2015-02-25T15:02:11-08:00] INFO: Processing file[/etc/ssl/chef_ca.cert] action create (/var/chef/cache/cookbooks/x509/providers/certificate.rb line 30)
[2015-02-25T15:02:11-08:00] INFO: file[/etc/ssl/chef_ca.cert] backed up to /var/chef/backup/etc/ssl/chef_ca.cert.chef-20150225150211.873229
[2015-02-25T15:02:11-08:00] INFO: file[/etc/ssl/chef_ca.cert] updated file contents /etc/ssl/chef_ca.cert
[2015-02-25T15:02:11-08:00] INFO: Chef Run complete in 2.903235 seconds

You can use openssl to verify your certificate:
openssl x509 -in /etc/ssl/chef.sacredsf.org.cert -issuer | head -1

issuer= /CN=ChefCA


Chef now provides a simpler way of generating client certificates than creating your own CA using openssl. However, there’s still some major pitfalls here that get in the way of a strong SSL client certificate solution.

For one thing, signing the CSRs is currently done manually. Ideally, I’d like to set up some kind of automatic script to sign the CSRs based on some policy, so clients can be approved without manual intervention.

Another issue is that because the CSRs require manual approval, it currently takes two chef-client runs to get an SSL cert. The next goal is to get this step down to being more automatic and streamlined.

Configuring Chef Server 12 to Use Trusted SSL Certs

Previously, I wrote on setting up Chef Server 12.

By default, Chef Server uses a self-signed certificate. This is fine for small testing purposes, but this becomes a significant problem in production, especially when using other Ruby tools or modules that require SSL verification.

Thus, I highly recommend configuring the Chef server to run with a real, trusted SSL certificate. For this example, I got a free one from StartSSL. I chose StartSSL because it’s free, and the root CAs are already in the default OS X and iOS trust stores. As in the previous article, this is with Chef Server 12 running on CentOS 6.6 in VMWare.

In this post, all of my examples will be for “chef.sacredsf.org”. Substitute in your own example.

NOTE: you do not need to have this server exist in DNS. As long as the server knows its own hostname, you can do this all on one VM.

Once you go through the StartSSL certificate wizard, you’ll be given four things (not all filenames will be accurate, I’ve renamed them for clarity):

  • The actual SSL certificate: chef_sacredsf_org_startssl.crt
  • The encrypted private key: chef_ssl_encrypted.key
  • The intermediate certificate: sub.class1.server.ca.pem
  • The root CA certificate: ca.pem

With all of these downloaded, here are the steps for setting up Chef to use these SSL certificates:

  1. First, decrypt the private key:
    openssl rsa -in chef_ssl_encrypted.key -out chef.sacredsf.org.nopassphrase.key
  2. Convert the .crt to a .pem file:
    openssl x509 -in chef_sacredsf_org_startssl.crt -out chef.sacredsf.org.pem -outform PEM
  3. Concatenate all the certs together into one .pem file:
    cat chef.sacredsf.org.pem <(echo) sub.class1.server.ca.pem <(echo) ca.pem > Complete/chef.sacredsf.org.pem

    (Note that I put this into a separate “Complete” folder to keep track of which one to copy)

  4. Copy Complete/chef.sacredsf.org.pem and chef.sacredsf.org.nopassphrase.key to your Chef server (via scp, or whatever mechanism works for you – since I did this in VMWare, I just used the drag-n-drop capability).
  5. Copy the key and certificate into the trusted store for CentOS 6:
    sudo cp chef.sacredsf.org.pem /etc/pki/tls/private/
    sudo cp chef.sacredsf.org.nopassphrase.key /etc/pki/tls/private/
  6. Add the certificate and key paths to /etc/opscode/chef-server.rb:

    Note: the last two parts, ssl_ciphers and ssl_protocols are optional – they just harden the SSL connection against weaker forms of SSL. You can leave them out and this will work just fine.

  7. IMPORTANT! Your server’s hostname must match the server in your certificate! Verify:
    hostname -f
    See https://www.centosblog.com/script-update-centos-linux-servers-hostname/ for more details on this.
  8. Finally, reconfigure the server with the new SSL certs:
    sudo chef-server-ctl reconfigure

If you have a problem here, or nginx takes too long to start up (or fails to start up), it’s almost certainly because of a problem with the certificates. Use cat to check the contents of the certs to make sure there are no typos, there is space between each BEGIN CERTIFICATE and END CERTIFICATE line (there should be 3 total), and that you copied the correct files.

You can verify your server’s SSL connection:
openssl s_client -connect chef.sacredsf.org:443

You should also verify in a web browser as well.

If all of it checks out and the web page loads, your SSL certs are working and your Chef server now uses trusted SSL!

Setting Up Chef Server 12 on CentOS 6 (and configuring an OS X client)

Chef is a configuration management tool, similar in concept to Puppet. As I test out various configuration management tools and methods, I wanted to try my hands at Chef to see what the process would be like.

I find that Chef’s documentation is a bit ambiguous, so I’ve recreated it here to show the steps I used to get something working.

The server / workstation setup was all done in a CentOS 6.6 VM.

The client configuration is done on an OS X Yosemite 10.10.2 VM.

Server Setup:

In these instructions, “ssh” refers to the short name for my organization “Schools of the Sacred Heart SF”.

Follow the instructions for Installing Chef Server:

  1. Download the Chef server for Enterprise Linux 6 – as of writing, that’s chef-server-core-12.0.4-1.el6.x86_64.rpm.
  2. Install the rpm:
    sudo rpm -Uvh chef-server-core-12.0.4-1.el6.x86_64.rpm
  3. If the hostname for the machine does not have an FQDN, that needs to be done first:
    hostname -f
  4. sudo chef-server-ctl reconfigure
  5. mkdir -p /etc/chef-server/
  6. Create your first user account:
    sudo chef-server-ctl user-create --filename /etc/chef-server/nick.pem nick Nick McSpadden nick.mcspadden@email.address password
  7. Create your first organization:
    sudo chef-server-ctl org-create ssh SacredSF --association_user nick --filename /etc/chef-server/ssh.pem

This “ssh.pem” file is the private key for my organization, with short name “ssh”. If you create an organization called “test” you can name the private key “test.pem” and use that wherever you see “ssh.pem” in these instructions.

Now, set up the Chef Workstation where you can get things done. This can be on the same computer, and in my case is in the same VM – the Server and Workstation are the same virtual machine. Follow instructions for Installing Chef Workstation:

  1. Download the Chef Workstation for Enterprise Linux 6 – as of writing, that’s chefdk-0.4.0-1.x86_64.rpm.
  2. Install the rpm:
    sudo rpm -Uvh chefdk-0.4.0-1.x86_64.rpm
  3. Do the initial setup:
    knife configure initial
    Answer the following questions:
Overwrite /home/nmcspadden/.chef/knife.rb? (Y/N) Y
Please enter the chef server URL: [https://chef:443] https://chef.domain.com:443/organizations/ssh
Please enter an existing username or clientname for the API: [admin] nick
Please enter the validation clientname: [chef-validator] ssh-validator
Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /etc/chef-server/ssh.pem
Please enter the path to a chef repository (or leave blank):

Set up the Chef Repo:

  1. git clone git://github.com/chef/chef-repo.git (I cloned this into my home directory, at ~/chef-repo/)
  2. mkdir -p chef-repo/.chef
  3. echo '.chef' >> ~/chef-repo/.gitignore
  4. Copy the two .pem files you created earlier into .chef:
    cp /etc/chef-server/*.pem ~/chef-repo/.chef/
  5. Write the file ~/chef-repo/.chef/knife.rb:
    log_level :info
    log_location STDOUT
    node_name 'nick'
    client_key '/home/nmcspadden/.chef/nick.pem'
    validation_client_name 'ssh-validator'
    validation_key '/etc/chef-server/ssh.pem'
    chef_server_url 'https://chef:443/organizations/ssh'
    syntax_check_cache_path '/home/nmcspadden/.chef/syntax_check_cache'
    cookbook_path '/home/nmcspadden/chef-repo/cookbooks'
  6. Run knife ssl fetch to trust the server’s self-signed cert.
  7. knife client list should now show you the name of your validator, which in this case is:
  8. Setup the proper Ruby paths and other services with Chef’s shell-initialization script:
    echo 'eval "$(chef shell-init bash"' >> ~/.bash_profile && source ~/.bash_profile

    Verify that ruby is correct using which ruby:

Node/Client Setup:

The “node” refers to an end client that is receiving Chef configurations. In this example, I’m using an OS X 10.10.2 VM as my client node.

  1. Add your Chef VM to /etc/hosts, if it doesn’t already exist in DNS.
  2. Install the Xcode Command Line Tools on the client machine first.
  3. Install the latest Chef client for OS X 10.10 – as of writing, that’s chef-12.0.3-1.dmg.
  4. Copy the ssh.pem created earlier to /etc/chef/validation.pem on the client (name is important).
  5. You need to copy the trusted certs from your chef-repo (~/chef-repo/.chef/trusted_certs/) to the client as well, into /etc/chef/trusted_certs/. If you don’t have these trusted certs on the chef server, use this command to generate them: knife ssl fetch
  6. Write a file client.rb to /etc/chef/client.rb:
  7. Run sudo chef-client to trigger the initial chef run.

You now have a working Chef client install with a working Chef server!