Incorporate Sal and JSS Data Into WebHelpDesk Inventory, using Docker

In previous posts, I covered:

WebHelpDesk, among its other features, makes a great inventory aggregate collector thanks to its use of discovery connections. Inventory data can be easily pulled from any flat database. Sal is a reporting engine for Munki that collects inventory data about OS X Munki clients, and JAMF Casper as an iOS MDM (referred to as Casper or “JSS” from here on out) stores inventory data about iOS clients.

We can set up scripts to pull data from Sal, using Sal-WHDImport, and from Casper, using JSSImport. This makes for a great triangle, allowing inventory aggregation into WebHelpDesk, and this is relatively trivial with Docker.

To save some time, I’ve incorporated the Sal-WHDImport script into Sal itself in a Dockerfile, available as the Sal-WHD container. We’ll be using this container below.

I’ve done the same thing with the JSSImport script, creating the JSSImport container.

Preparing Data Files:

Sal requires some modification in order to talk to WebHelpDesk. We’re going to use a plugin Graham Gilbert wrote called WHDImport to sync the Sal data into a single flat database for WebHelpDesk to pull from.

First, we’ll need to modify On the Docker host:

  1. mkdir -p /usr/local/sal_data/settings/
  2. curl -o /usr/local/sal_data/settings/

Make the following changes to
Add 'whdimport', (with the comma) to the end of the list of INSTALLED_APPS.

Next, we’ll clone a copy of MacModelShelf:
git clone /usr/local/sal_data/macmodelshelf

MacModelShelf was originally developed by Per Oloffson, but this version is my fork that uses a JSON database, which seems to improve cross-platform compatibility. The purpose of cloning a local copy is to keep the JSON database, which is automatically populated with model lookups. By keeping a local copy, we can safely spin up and down WebHelpDesk containers without losing any of our lookup data (which may save milliseconds in future lookups).

Run the Sal DB and Setup Scripts:

First, we create a data-only container for Sal’s Postgres database, and then run the Postgres database. We can specify all the variables at runtime using the -e arguments. The only thing you’ll need to change below is the password.

  1. docker run --name "sal-db-data" -d --entrypoint /bin/echo grahamgilbert/postgres Data-only container for postgres-sal
  2. docker run --name "postgres-sal" -d --volumes-from sal-db-data -e DB_NAME=sal -e DB_USER=saldbadmin -e DB_PASS=password --restart="always" grahamgilbert/postgres

Run the JSS Import DB:

We do the same thing with the JSS Import container’s database. Again, change the password only. Note that we’re using a slightly different Postgres container for this – the macadmins/postgres instead of grahamgilbert/postgres.

  1. docker run --name "jssi-db-data" -d --entrypoint /bin/echo macadmins/postgres Data-only container for jssimport-db
  2. docker run --name "jssimport-db" -d --volumes-from jssi-db-data -e DB_NAME=jssimport -e DB_USER=jssdbadmin -e DB_PASS=password --restart="always" macadmins/postgres

Run the WHD DB:

There’s a theme here – change the password for WebHelpDesk’s Postgres database.

  1. docker run -d --name whd-db-data --entrypoint /bin/echo macadmins/postgres Data-only container for postgres-whd
  2. docker run -d --name postgres-whd --volumes-from whd-db-data -e DB_NAME=whd -e DB_USER=whddbadmin -e DB_PASS=password --restart="always" macadmins/postgres

Run Temporary Sal to Prepare Initial Data Migration:

Load a temporary container just for the purpose of setting up Sal’s Django backend to incorporate the WHDImport addition.

Note that we’re using --rm with this docker run command, because this is intended only to be a transient container for the purpose of setting up the database. It will remove itself when complete, but the changes to the database will be permanent.

docker run --name "sal-loaddata" --link postgres-sal:db -e ADMIN_PASS=password -e DB_NAME=sal -e DB_USER=saldbadmin -e DB_PASS=password -it --rm -v /usr/local/sal_data/settings/ macadmins/salwhd /bin/bash

This opens a Bash shell. From that Bash shell:

  1. cd /home/docker/sal
  2. python syncdb --noinput
  3. python migrate --noinput
  4. echo "TRUNCATE django_content_type CASCADE;" | python dbshell | xargs
  5. python schemamigration whdimport --auto
  6. python migrate whdimport
  7. exit
  8. After exiting, the temporary “sal-loaddata” container is removed.

Run Sal and Sync the Database:

Load up the Sal container and run “syncmachines” to get started. Change the passwords here to match what you used previously:

  1. docker run -d --name sal -p 80:8000 --link postgres-sal:db -e ADMIN_PASS=password -e DB_NAME=sal -e DB_USER=saldbadmin -e DB_PASS=password -v /usr/local/sal_data/settings/ --restart="always" macadmins/salwhd
  2. docker exec sal python /home/docker/sal/ syncmachines

Run JSSImport and Sync the Database:

Run the JSSImport container, which will pull the device list from Casper and sync it into the jssimport database.

If you haven’t already, set up an API-only user account in the JSS, and use those credentials below. Change the URL to match your Casper instance.

docker run --rm --name jssi --link jssimport-db:db -e DB_NAME=jssimport -e DB_USER=jssdbadmin -e DB_PASS=password -e JSS_USER=user -e JSS_PASS=password -e JSS_URL= --restart="always" macadmins/jssimport

Although I haven’t tested this particular permutation, you could theoretically build a JSS Docker instance, and then link it to the jssimport container (--link jss:jss), and just use the URL -e JSS_URL=https://casper.

Run WHD with its data-only container:

Now run WebHelpDesk with its linked databases.

  1. docker run -d --name whd-data --entrypoint /bin/echo macadmins/whd Data-only container for whd
  2. docker run -d -p 8081:8081 --link postgres-sal:saldb --link postgres-whd:db --link jssimport-db:jdb --name "whd" --volumes-from whd-data --restart="always" macadmins/whd

WebHelpDesk now has direct access to three linked databases – its own Postgres database, as db; the Sal database, known as saldb; and the JSS Import database, known as jdb. This will make it trivially easy to pull the data it needs.

Configure WHD Through Browser:

  1. Open your web browser on the Docker host: http://localhost:8081
  2. Set up using Custom SQL Database:
    1. Database type: postgreSQL (External)
    2. Host: db
    3. Port: 5432
    4. Database Name: whd
    5. Username: whddbadmin
    6. Password: password
  3. Skip email customization
  4. Setup administrative account/password
  5. Skip the ticket customization

Setup Discovery Connections:

In WebHelpDesk, go to Setup > Assets > Discovery Connections. Make your two connections for Sal and the JSS.

  1. Setup discovery disconnection “Sal”:
    1. Connection Name: “Sal” (whatever you want)
    2. Discovery Tool: Database Table or View
    3. Database Type: PostgreSQL – uncheck Use Embedded Database
    4. Host: saldb
    5. Port: 5432
    6. Database Name: sal
    7. Username: saldbadmin
    8. Password: password
    9. Schema: Public
    10. Table or View: whdimport_whdmachine
    11. Sync Column: serial
  2. Setup discovery connection “Casper”:
    1. Connection Name: “Casper” (whatever you want)
    2. Discovery Tool: Database Table or View
    3. Database Type: PostgreSQL – uncheck Use Embedded Database
    4. Host: jdb
    5. Port: 5432
    6. Database Name: jssimport
    7. Username: jssdbadmin
    8. Password: password
    9. Schema: Public
    10. Table or View: casperimport
    11. Sync Column: serial

Now, you have a single web service that handles all inventory collection.

From here, if you wanted to schedule this for automation, you’d only need to run these two tasks regularly:

  1. docker exec sal python /home/docker/sal/ syncmachines (since the sal container is daemonized and runs persistently).
  2. docker run --rm --name jssi --link jssimport-db:db -e DB_NAME=jssimport -e DB_USER=jssdbadmin -e DB_PASS=password -e JSS_USER=user -e JSS_PASS=password -e JSS_URL= macadmins/jssimport (since this container is a fire-and-forget container that self-deletes on completion).

You could set up a crontab to run those two tasks nightly, and then set up WebHelpDesk’s internal syncs to its discovery connections to occur just an hour or so afterwards.

Once your inventory data is aggregated, you could use other tools like my WHD-CLI script to access WebHelpDesk via a Python interpreter, allowing for more scriptability. This is also available in a Docker container.

Using WHD-CLI, you have instant scriptable access to your inventory system, which could be used for lots of neat things, including a way to guarantee that a Puppetmaster only signs approved devices. Lots to explore!

Enhancing Sal with Facter and Profiles

In a previous post, I showed how to set up Sal.

Sal’s basic functionality is useful on its own, for the basic Munki reporting – what are the completed installs, pending updates, what OS versions, how many devices checked in the past 24 hours, etc. In this post, I’m going to demonstrate how to get more out of Sal.

Adding in Facter:

You can add much more, though, by the use of Puppet, and more specifically, the piece of Puppet called Facter. Facter is a separate program that works with Puppet that simply gathers information (“facts”) about the host OS and stores them (ostensibly so that Puppet can determine what the machine’s state is and what needs to happen to it to bring it in line with configured policy).

At the bottom of Sal’s client configuration guide is a small section on using custom Facter facts. Puppet is not required to use Facter, and you can actually download it yourself as part of Puppet’s open source software.

Note: if you’re an Autopkg user, you can find a Facter recipe in the official default repo: autopkg run (or autopkg run Facter.munki if you have Munki configured with Autopkg).

Install Facter on your clients, either with Munki or by simply installing the Facter.pkg.

Test out Facter on that client by opening up the Terminal and running Facter:
You’ll see a whole lot of information about the client printed out. Handy!

Additional Facts:

A nice thing about Facter is that it’s easy to extend and customize with additional facts, which are essentially just Ruby scripts. Puppet Labs has documentation on Custom Facts here.

Graham Gilbert, the author of Sal, has also written some helpful custom facts for Macs, which I’m going to use here.

We’re going to need to get these facts downloaded and onto our clients. Use whatever packaging utility you like to do this, but all of those .rb files have to go into Facter’s custom facts directory. There are lots of places to put them, but I’m going to place them in /var/lib/puppet/lib/facter/, where they can also be used by Puppet in the future.

Once those facts are installed on your client, you can run Facter again and access them using an additional argument:
sudo facter --puppet

Note that you now need sudo to see these extra facts. Facter needs administrative privileges to get access to those facts, so simply running facter --puppet will give you the same results we had previously (before the new Mac facts were installed). This won’t be a problem, as the Sal postflight script, when executed with Munki, is run as root.

To make use of Facter with Sal, we need only run Munki again, which executes the Sal postflight:
sudo managedsoftwareupdate

When the run is complete, take a look at the machine’s information in Sal. You’ll now see a “Facter” toggle with all of those neat facts for that client machine.

Faster Client Configuration:

One of the instructions as part of my Sal setup post, as well as part of the official documentation, is to set the client’s preferences for Sal URL and the Machine Group key for it to use. This was done using the defaults command to write the preferences to the com.salsoftware.sal preferences domain.

Instead of using defaults at the command line, we could also provide a simple .plist file that contains the two keys (machine group key, and URL) and two values, and place that in /Library/Preferences/com.salsoftware.sal.plist. However, relying on .plist files to load preferences is problematic with cfprefsd, the preference caching daemon introduced in 10.9 Mavericks.

Well, if you can do it with defaults, you can do it with configuration profiles! Configuration profiles (also known as .mobileconfig files) allow us to enforce preference domain values – such as enforcing the key and URL values for com.salsoftware.sal.

Making a configuration profile by hand is madness, so it’s better to use a tool that already produces profiles effectively – such as Profile Manager, Apple Configurator, or any MDM suite. That’s a lot of work just to get a profile, though.

Instead, we can thank Tim Sutton for his awesome mcxToProfile script, which takes a .plist or existing MCX object and converts it into a profile. We could use mcxToProfile to convert an existing com.salsoftware.sal.plist into a profile, but that means we now need to handcraft a .plist file for each Machine Group key we create.

I’m not a fan of manual tasks. I’m a big fan of automation, and I like it when we make things as simple, automatic, and repetitive as possible. We want a process that will do the same thing every time. So rather than create a plist for each Machine Group I want a profile for, and then run the mcxToProfile script, I’m going to write another script that does it for me.

All of this can be found on my Github repo for SalProfileGenerator.

Writing the script:

Here’s the code for the script:


import argparse
import os
from mcxToProfile import *

parser = argparse.ArgumentParser()
parser.add_argument("key", help="Machine Group key")
parser.add_argument("-u", "--url", help="Server URL to Sal. Defaults to http://sal.")
parser.add_argument("-o", "--output", help="Path to output .mobileconfig. Defaults to 'com.salsoftware.sal.mobileconfig' in current working directory.")
args = parser.parse_args()

plistDict = dict()

if args.url:
	plistDict['ServerURL'] = args.url
	plistDict['ServerURL'] = "http://sal"

plistDict['key'] = args.key

newPayload = PayloadDict("com.salsoftware.sal", makeNewUUID(), False, "Sal", "Sal")

newPayload.addPayloadFromPlistContents(plistDict, 'com.salsoftware.sal', 'Always')

filename = "com.salsoftware.sal"

filename+="." + plistDict['key'][0:5]

if args.output:
	if os.path.isdir(args.output):
		output_path = os.path.join(args.output, filename + '.mobileconfig')
	elif os.path.isfile(args.output):
		output_path = args.output
		print "Invalid path: %s. Must be a valid directory or an output file." % args.output
	output_path = os.path.join(os.getcwd(), filename + '.mobileconfig')


Looking at the script, the first thing we see is that I’m importing mcxToProfile directly. No need to reinvent the wheel when someone else already has a really nice wheel with good tires and spinning rims that is also open-source.

Next, you see the argument parsing. As described in the README, this script takes three arguments:

  • the Machine Group key
  • the Sal Server URL
  • the output path to write the profiles to

The payload of each profile needs to be enforced settings for com.salsoftware.sal, with the two settings it needs – the key and the URL. The URL isn’t likely to change for our profiles, so that’s an easy one.

First, initialize mcxToProfile’s PayloadDict class with our identifier (“com.salsoftware.sal”), a new UUID, and filler content for the Organization, etc. We call upon mcxToProfile’s addPayloadFromPlistContents() function to add in “always” enforcement of the preference domain com.salsoftware.sal.

The obvious filename to use for our profile is “com.salsoftware.sal.mobileconfig”. This presents a slight issue, because if our goal is to produce several profiles, we can’t name them all the same thing. The simple solution is to take a chunk of the Machine Group key and throw it into the filename – in this case, the first 5 letters.

Once we determine that our output location is valid, we can go ahead and save the profile.

Ultimately we should get a result like this:

./ e4up7l5pzaq7w4x12en3c0d5y3neiutlezvd73z9qeac7zwybv3jj5tghhmlseorzy5kb4zkc7rnc2sffgir4uw79esdd60pfzfwszkukruop0mmyn5gnhark9n8lmx9
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">
	<string>Included custom settings:

Git revision: a9edc21c62</string>

Adjusting mcxToProfile:

On OS X, plists can be handled and parsed easily because it’s built into the Foundation libraries. mcxToProfile itself incorporates several functions from Greg Neagle’s FoundationPlist library, which does improved plist handling compared to Python’s built-in plistlib.

Because of the reliance on the OS X Foundation libraries, however, we can’t use FoundationPlist outside of OS X. Since Sal is built to run on multiple platforms, and the Docker image is built on Ubuntu, we can’t use FoundationPlist as the core of our plist handling functionality.

Thus, we’ll need to make some adjustments to mcxToProfile:

	from FoundationPlist import *
	from plistlib import *

In Tim Sutton’s original version of the script, he imports the necessary Foundation libraries into Python for use of them, and inline copied the parts of FoundationPlist he needed. If we’re going to make this more cross-platform friendly, we need to remove those dependencies.

So in my revision of mcxToProfile, I’ve removed all of the FoundationPlist functions completely from the code, instead relying on bundling a copy of with the project. Instead of importing Foundation libraries, we’re going to try to use FoundationPlist – and if any part of that import goes wrong, we just abandon the whole thing and use Python’s built-in plistlib.

Dirty, but effective, and necessary for cross-platform compatibility.

Now we have a simple script for generating a profile for a Machine Key and URL for Sal that can run on any platform.

Automating the script:

Generating a single profile is a useful first step. The ultimate goal is to be able to generate all of the profiles we’ll need at once.

This script was written in Bash, rather than Python. You can find it in the Github repo here:


profile_path=`printenv PROFILE_PATH`
if [[ ! $profile_path ]]; then

results=$( echo "SELECT key FROM server_machinegroup;" | python /home/docker/sal/ dbshell | xargs | awk {'for (i=3; i<NF-1; i++) print $i'} )
read -rd '' -a lines <<<"$results"
for line in "${lines[@]}"
	if [[ -z $1 ]]; then
		/usr/local/salprofilegenerator/ $line --output $profile_path
		/usr/local/salprofilegenerator/ $line --url $1 --output $profile_path

It’s ugly Bash, I won’t deny. The README documents the usage of this script in detail.

The assumption is that this will be used within the Sal Docker container, and thus we can make use of environment variables. With that assumption, I’m also expecting that an environment variable PROFILE_PATH gets passed in that can be used as the location to place our profiles. Absent the environmental variable, I chose /home/docker/profiles as the default path.

The purpose of the IFS here is to help parse a long string based on newlines.

The actual pulling of the machine group keys is the complex part. I’m going to break down that one liner a bit:
echo "SELECT key FROM server_machinegroup;"
This is the SQL command that will give us the list of machine group keys from the Postgres database.

python /home/docker/sal/ dbshell
This invokes the Django script to open up a database shell, which allows us to execute database commands directly from the command line. Since dbshell opens up an interpreter, we’re going to pipe standard input to it by echoing the previous SQL query.

Without going into a huge amount of unnecessary detail about xargs, the purpose of this is simply to compress the output into a single line, rather than multiple lines, for easier parsing.

awk {'for (i=3; i<NF-1; i++) print $i'}

Pretty much any time I start using awk in Bash, you know something has gone horribly wrong with my plan and I should probably have just used Python instead. But I didn’t, so now we’re stuck here, and awk will hopefully get us out of this mess.

In a nutshell, this awk command prints all the words starting at 4 through the last-word-minus-two. Since dbshell queries produces output at the end saying how many rows were produced as a result, we need to skip both the number and the word “rows” at the very end. The parsing works out because we set the IFS to divide words up based on newlines.

Ultimately, this handles the odd formatting from dbshell and prints out just the part we want – the two Machine Group keys.

read -rd '' -a lines <<<"$results"

This takes the list of Machine Group keys produced by the long line and shoves it into a Bash array.

for line in "${lines[@]}"
The for loop iterates through the array. For each key found in the array, call the script.

As the README documents, the shell script does handle a single shell argument, if you want to pass a different URL than the default. If a shell argument is found, that is used as a --url argument to the script.

By calling the script, we now get a .mobileconfig profile for each Machine Group key. Those profiles can be copied off the Sal host (or out of the Sal container) and into a distribution system, such as Profile Manager, an MDM, or Munki. Installing profiles on OS X is a trivial matter, using the profiles command or simply double-clicking them and installing them via GUI.

Because I’m currently in a “dockerize ALL THE THINGS” phase right now, I went ahead and created a Docker image for Sal incorporating this profile generation script.


Munki is a very useful tool. Munki with Sal by itself is a useful tool, but the best tools are ones that can be extended. Munki, Sal, and Facter provide great information about devices. Making Sal easy to install lessens the burden of setting it up, and makes the entire process of migrating to a more managed environment simpler.

Setting Up Sal for Munki Reporting

Munki is an incredible tool for Mac deployment. Unlike other MDM or management software, it’s focused on a single purpose – delivering files from a central repository to a client machine based on some criteria. It doesn’t include features like inventory tracking, reporting, queries, or other items that you might find in commercial management solutions and suites.

Since Munki is open source and has a thriving community of dedicated users, it’s no surprise that solutions have been developed to add this kind of functionality to Munki. There are a number of options out there, but the one I’m going to focus on is the open-source version of Sal, written by Graham Gilbert.

Sal is Django-based web app that collects information from Munki clients whenever they run the Munki software. It allows for convenient access to inventory collection, which can give us an idea of what OSes we’re seeing on our clients, what software packages are installed, what updates are still pending, etc. Information and reporting is always good, and Sal does a great job.

Sal has solid documentation for using it and getting started, so I won’t reproduce all of that here. Instead, I’m just going to go through a simple setup of the Sal Docker container and installation on client devices, from start to finish.

Getting Sal running with Docker:

As suggested by the repo instructions, the Docker image is the officially recommended approach for setting up Sal.

First, you’ll need to get Graham’s customized Postgres database running. His customization allows for easy database creation by passing in environmental variables, which I used in a previous blog post about customizing Postgres.

I prefer to use data containers, to keep my data portable and not tied to a host. Here’s the data container for Sal’s Postgres database:
docker run --name "sal-db-data" -d --entrypoint /bin/echo grahamgilbert/postgres Data-only container for postgres-sal

Then the database:
docker run --name "postgres-sal" -d --volumes-from sal-db-data -e DB_NAME=sal -e DB_USER=saldbadmin -e DB_PASS=password --restart="always" grahamgilbert/postgres

The -e environment variables allow us to specify the database name, user, and password for access.

Now run Sal itself:
docker run -d --name sal -p 80:8000 --link postgres-sal:db -e DOCKER_SAL_TZ="America/Los_Angeles" -e ADMIN_PASS=password -e DB_NAME=sal -e DB_USER=saldbadmin -e DB_PASS=password macadmins/sal

Specify real passwords for use in production, obviously. I’ve also passed in the DOCKER_SAL_TZ timezone environmental variable to change it to PST, since I don’t live in London.

Configure Sal:

Open your web browser to http://localhost/ on the Docker host to log into Sal – using the password you specified earlier.

Create a Business Unit.

Create at least one Machine Group. Each Machine Group will generate a “key,” which you’ll need to add to the clients.

Set up the clients:

Install the provided Sal-scripts.pkg onto an OS X client with Munki installed.

If Sal has not been added to your DNS (if you’re testing this for the first time, this will almost certainly be true), you’ll need to modify /etc/hosts on the client to add in your Docker host as “sal”.

Next, you’ll need to add the proper client configuration to your OS X clients.

Change the URL to http://sal for this example, but you’ll need to set the key to the one of the keys you generated from a Machine Group:

defaults write /Library/Preferences/com.salsoftware.sal ServerURL http://sal
defaults write /Library/Preferences/com.salsoftware.sal key e4up7l5pzaq7w4x12en3c0d5y3neiutlezvd73z9qeac7zwybv3jj5tghhmlseorzy5kb4zkc7rnc2sffgir4uw79esdd60pfzfwszkukruop0mmyn5gnhark9n8lmx9

Now, you can simply run Munki’s managedsoftwareupdate to get Sal reporting:
sudo managedsoftwareupdate

At then end of the Munki run, you should see output similar to this:

    Performing postflight tasks...
    postflight stdout: 
Sal report submmitted for Mac.local.