Puppet Catalog Diff

Table of Contents

1. Overview
2. Module Description
3. Setup
   1. Set up node discovery
   2. Set up auth.conf
4. Usage
   1. Multi threaded compile requests
   2. Fact search
   3. Changed depth
   4. Output report
5. Limitations
6. Previous Authors
7. Contributors
8. See Also
   1. Upload facts to PuppetDB
   2. Modern fact submission

Overview

A tool to compare two Puppet catalogs.

Module Description

While upgrading versions of Puppet or refactoring Puppet code you want to ensure that no unexpected changes will be made prior to committing the changes.

This tool will allow you to diff catalogs created by different versions of Puppet or different environments. This will let you gauge the impact of a change before actually touching any of your nodes.

This tool is delivered as a collection of Puppet Faces. It thus requires a Puppet 6.11 (or newer) installation to properly run.

Only the system that runs catalog-diff needs to be using Puppet 6.11 or newer. The puppetservers you are targeting to do the catalog compilation can be running an early version of puppet; however puppetdb must be at least version 2.3. You can use catalog-diff directly on a Puppetserver but also on another server.

The diff tool recognizes catalogs in yaml, marshall, json, or pson formats. Currently automatic generation of the catalogs is done in the pson format.

The tool can automatically compile the catalogs for both your new and older servers/environments. It can ask the master to use PuppetDB to compile the catalog for the last known environment with the last known facts. It can then validate against PuppetDB that the node is still active. This filtered list should contain only machines that have not been decommissioned in PuppetDB (important as compiling their catalogs would also reactive them and their exports otherwise).

When you are comparing between different versions of Puppet using two Master servers you are going to need to copy facts from the old Master to the new one in order to be able to compile catalogs on the new Master. This is useful when upgrading Puppet version.

To upload facts to PuppetDB on a Master see the Upload facts script.

Setup

Set up node discovery

Node discovery requires an access to the PuppetDB. You'll need either:

• have an unencrypted access to PuppetDB (port 8080, local or proxified)
• generate a set key and certificate signed by the Puppet CA to access the PuppetDB

PuppetDB has an (optional) allowlist for certificates that are allowed to connect to the database. It's located at /etc/puppetlabs/puppetdb/certificate-allowlist. in Puppet Enterprise you can configure it like this to allow a specific certificate:

puppet_enterprise::profile::puppetdb::allowlisted_certnames:
  - catalog-diff

Set up auth.conf

Once you have set up the discovery, you need to allow access to the "diff" node to compile the catalogs for all nodes on both your old and new masters.

On Puppet 5+, you need to edit the Puppetserver's /etc/puppetlabs/puppetserver/conf.d/auth.conf file.

In your confdir modify auth.conf to allow access to /catalog. If there is an existing reference i.e. the $1 back reference for machines to compile their own catalog then simply add another line with the certificate name of the diff machine. As mentioned this can be the new master as required.

E.g. if you're using Puppet 5, you should have something like:

{
    # Allow nodes to retrieve their own catalog
    match-request: {
        path: "^/puppet/v3/catalog/([^/]+)$"
        type: regex
        method: [get, post]
    }
    allow: ["$1","catalog-diff"]
    sort-order: 500
    name: "puppetlabs catalog"
},

If you are on Puppet 6, you can activate the certless API instead with:

{
    match-request: {
        path: "^/puppet/v4/catalog"
        type: regex
        method: [post]
    }
    allow: ["catalog-diff"]
    sort-order: 500
    name: "puppetlabs certless catalog"
},

You can update the auth.conf with the following Puppet code (uses the puppetlabs/puppet_authorization module):

puppet_authorization::rule { 'catalog-diff certless catalog':
  match_request_path   => '^/puppet/v4/catalog',
  match_request_type   => 'regex',
  match_request_method => 'post',
  allow                => 'catalog-diff',
  sort_order           => 500,
  path                 => '/etc/puppetlabs/puppetserver/conf.d/auth.conf',
}

The certless API has one big, not so obvious, advantage. It can read trusted facts from PuppetDB and use them during catalog compilation. Using trusted facts in Hiera/Puppet code required using the certless API. From the API docs:

trusted_facts (API field): A hash with a required values key containing a hash of the trusted facts for a node. In a normal agent's catalog request, these would be extracted from the cert, but this endpoint does not require a cert for the node whose catalog is being compiled. If not provided, Puppet will attempt to fetch the trusted facts for the node from PuppetDB or from the provided facts hash.

Usage

Example: diff catalogs for node1.example.com & node2.example.com between puppetserver puppet5.example.com, puppet6.example.com. The old catalog will be fetched from PuppetDB, the new one will be compiled:

$ puppet module install puppet-catalog_diff
$ puppet catalog diff \
     puppet5.example.com:8140/production puppet6.example.com:8140/production \
     --filter_old_env \
     --old_catalog_from_puppetdb \
     --certless \
     --show_resource_diff \
     --content_diff \
     --ignore_parameters alias \  # Puppet6 removes lots of alias parameters
     \ #--yamldir $YAMLDIR \
     \ #--ssldir $SSLDIR \
     --changed_depth 1000 \
     --configtimeout 1000 \
     --output_report "${HOME}/lastrun-$$.json" \
     --debug \
     \ #--fact_search kernel='Darwin' \
     --threads 50 \
     \ #--node_list=node1.example.com,node2.example.com

Example: Compare to local catalogs for node1.example.com (we recommend absolute paths):

$ puppet catalog diff /foo/old/node1.example.com.json /foo/new/node1.example.com.json

You can generate them on an agent in a serverless setup:

puppet catalog compile --render-as json

As an alternative an agent can also download its catalog and store it locally:

puppet catalog download

Multi threaded compile requests

You can change the number of concurrent connections to the masters by passing an interger to the --threads option. This will balence the catalogs evenly on the old and new masters. This option defaults to 10 and in testing 50 threads seemed correct for 4 masters with two load balancers.

Note: When using catalog diff to compare directories, one thread per catalog comparison will be created. However, since Ruby cannot take advantage of multiple CPUs this may be of limited use comparing local catalogs. If the 'parallel' gem is installed, then one process will be forked off per CPU on the system, allowing use of all CPUs.

Fact search

You can pass --fact_search to filter the list of nodes based on a single fact value. This currently defaults to kernel=Linux if you do not pass it. This query will be passed as a filter to the PuppetDB to retrieve the list of nodes to compare.

Node list

Passing --node_list will bypass the dynamic generation of node lists from PuppetDB including the --fact_search filter. The list of nodes are not validated against PuppetDB, and it is up to the user to ensure that the nodes exist and are active.

Changed depth

Once each catalog is compiled , it is saved to the /tmp directory on the system and the face will then automatically calculate the differences between the catalogs. Once this is complete a summary of number of nodes with changes as well as nodes whose catalog would not compile are listed. You can modify the number of nodes shown here using --changed_depth option.

Output Report

You can save the last report as json to a specific location using "--output_report" This report will contain the structured data in the format of running this command with --render-as json. An example Rakefile is provided with a docs task for converting this report to (GitHub flavored) markdown. The script above also will save the output with escaped color. If you want to view that text report run less -r lastrun-$$.log

Non-default PuppetDB/Configuring PuppetDB

Usually, Puppet uses its default PuppetDB. This is configured in the puppetdb.conf. The file is located at $(puppet config print confdir)/puppetdb.conf (usually /etc/puppetlabs/puppet/puppetdb.conf). It's present by default on all Puppetservers that talk to a PuppetDB. puppet-catalog-diff will use the first entry in that file.

It's recommended to run puppet-catalog-diff as a normal user, not as root user. In that case the confdir is different and you need to create the puppetdb.conf explicitly. It's a simple ini format:

[main]
server_urls = https://fqdn:8081

You can even run puppet-catalog-diff as non-root on a system that's not a Puppetserver. In that case you need to install puppetdb-termini in addition to the Puppet Agent.

The spec allows you to list multiple PuppetDBs in the puppetdb.conf, however puppet-catalog-diff always uses the first entry. By default, this server will be used to discover nodes in PuppetDB, to get old catalogs from PuppetDB and to get new catalogs.

If you like, you can provide two explicit PuppetDB URIs for different purposes. For node discovery and retrieving old catalogs, you can use --old_puppetdb https://fqdn:8081. To get new catalogs from a specific Puppetdb, use --new_puppetdb https://fqdn:8081.

puppet catalog diff works with the TLS certificates that the agent also uses. You can see the related files by checking puppet config print | grep ssl. If the old PuppetDB uses certificates from a different CA, you can provide those via CLI options Those are:

• --old_puppetdb_tls_cert=
• --old_puppetdb_tls_key=
• --old_puppetdb_tls_ca=

Non-default Puppetserver

puppet catalog diff can request an old catalog from a Puppetserver. The Puppetserver will compile a new catalog. By default, catalog-diff will use the Agent default certificates to connect to the old Puppserver (see the section above for details). You can provide custom Client TLS certificate/private key and a CA file:

• --old_puppetserver_tls_cert=
• --old_puppetserver_tls_key=
• --old_puppetserver_tls_ca=

Limitations

This code only validates the catalogs, it cannot tell you if the behavior of the providers that interpret the catalog has changed so testing is still recommended, this is just one tool to take away some of the uncertainty.

You can get some inline help with:

puppet man catalog

The reports generated by this tool can be rendered as json as well as viewed in markdown using the Rakefile in this directory. A web viewer is also available at https://github.com/voxpupuli/puppet-catalog-diff-viewer

Previous Authors

R.I.Pienaar rip@devco.net / www.devco.net / @ripienaar Zack Smith zack@puppetlabs.com / @acidprime Raphaël Pinson raphael.pinson@camptocamp.com / @raphink

Contributors

The list of contributors can be found at: https://github.com/voxpupuli/puppet-catalog_diff/graphs/contributors.

See also

Upload facts to PuppetDB

Standalone Ruby script upload_facts.rb that is used to upload yaml files with facts to PuppetDB. This is useful when you are upgradering Puppet version and uses two different Puppet Masters for this. Then you can use this script to upload facts from the old Master to the new one. The script can also be used to just refresh the facts in PuppetDB from the old Master. These facts are required to be able to compile the catalogs on the new Master.

The script uses yaml-files in the same format as stored on the Puppet Master when real agents report their facts at the beginning of a Puppet Agent execution.

The script is developed to be executed on the Puppet Master, so the yaml-facts files should be copied to the new Master using scp or similar, preferably to the $(puppet config print vardir)/yaml/facts directory.

Then all files in the directory can be uploaded to PuppetDB by using this command:

$ ./upload_facts.rb $(puppet config print vardir)/yaml/facts/*.yaml

The script is available at https://github.com/JohnEricson/upload_facts.

It's been verified to work with uploading facts from Puppet Masters running Puppet version 3 to Masters running version 5. It uses the /puppet/v3/facts/ API which is available in version 3 and >= 5 of Puppet. This API was removed in Puppet 4 but added again in 5.

Modern fact submission

Nowadays it's possible to use puppet facts upload --server $new_server to submit facts to a new server. This however requires that the new puppetserver and the old one share one certificate authority. You can easily run this once via bolt to get all facts to a new puppetserver.

complex fact submission

To every problem an overengineered solution exists! Let's assume this: You have an existing Puppet environment. You setup a new Puppetserver, with a newer Puppet version and a new CA. You have a third box with catalog_diff, that has certificates to access the old and new Puppetserver. Now for catalog_diff to work, we need to get the facts from the old environment to the new one. There are three little scripts that you can use to:

• download facts from old PuppetDB
• Convert the format
• Submit them to the new Puppetserver

It's best to run them on the catalog_diff box, since it already has certificates that allow it to access all required APIs:

#!/bin/bash


#differ_certs/
#├── catalog-diff_dev
#│   ├── ca
#│   │   └── ca.pem
#│   ├── cert
#│   │   └── catalog-diff.pem
#│   └── private
#│       └── catalog-diff.pem
#├── catalog-diff_prod
#│   ├── ca
#│   │   └── ca.pem
#│   ├── cert
#│   │   └── catalog-diff.pem
#│   └── private
#│       └── catalog-diff.pem


certs_dir="${HOME}/differ_certs"
certs_dev="${certs_dir}/catalog-diff_dev"
certs_prod="${certs_dir}/catalog-diff_prod"
cert='catalog-diff.pem'
puppetdb_dev=puppet-dev.local
puppetdb_prod=puppet-prod.local
clientcert_dev="${certs_dev}/cert/${cert}"
clientcert_prod="${certs_prod}/cert/${cert}"
clientkey_dev="${certs_dev}/private/${cert}"
clientkey_prod="${certs_prod}/private/${cert}"
cacert_dev="${certs_dev}/ca/ca.pem"
cacert_prod="${certs_prod}/ca/ca.pem"

function prod_facts() {
  curl --request GET \
    --url "https://${puppetdb_prod}:8081/pdb/query/v4/factsets" \
    --cert "${clientcert_prod}" \
    --cacert "${cacert_prod}" \
    --key "${clientkey_prod}" \
    --silent \
    | jq -cr '.[] | .certname, .' | awk 'NR%2{f="factsets/"$0".json";next} {print >f;close(f)}'
}

function dev_facts() {
  curl --request GET \
    --url "https://${puppetdb_dev}:8081/pdb/query/v4/factsets" \
    --cert "${clientcert_dev}" \
    --cacert "${cacert_dev}" \
    --key "${clientkey_dev}" \
    --silent \
    | jq -cr '.[] | .certname, .' | awk 'NR%2{f="factsets/"$0".json";next} {print >f;close(f)}'
}

function facts() {
  dev_facts
  prod_facts
}
facts

#!/opt/puppetlabs/puppet/bin/ruby

require 'json'
require 'date'

Dir[Dir.home + "/factsets/*.json"].each do |file|
  filename = File.basename(file)
  puts "processing #{filename}"
  facts = JSON.parse(File.read(file))
  real_facts = { }
  real_facts['values'] = facts['facts']['data'].map{|facthash| {facthash['name'] => facthash['value']}}.reduce({}, :merge)
  real_facts['name'] = facts['certname']
  real_facts['timestamp'] = facts['timestamp']
  # expiration is usually timestamp + runintervall. We use 30min here
  real_facts['expiration'] = DateTime.parse(facts['timestamp']) + Rational(30 * 60, 86400)
  File.open(Dir.home + "/facts/#{filename}","w") do |f| f.write("#{JSON.pretty_generate(real_facts)}\n") end
end

#!/bin/bash

hostcert="$(puppet config print hostcert)"
hostprivkey="$(puppet config print hostprivkey)"
localcacert="$(puppet config print localcacert)"
server="$(puppet config print server)"
for file in facts/*json; do
  filename="$(basename $file)"
  certname="$(basename $filename '.json')"
  environment="$(jq --raw-output .environment factsets/${filename})"
  curl --include \
    --request PUT \
    --cert "${hostcert}" \
    --key "${hostprivkey}" \
    --cacert "${localcacert}" \
    --data @"${file}" \
    --url "https://${server}:8140/puppet/v3/facts/${certname}?environment=${environment}" \
    --header 'Content-Type: application/json'
done

Further documentation

• Raphaël Pinson wrote a blog series on dev.to about using puppet-catalog-diff and GitLab integration
• Raphaël Pinson also made two talks about it:
  • https://youtu.be/6LOaHsQDsiI - Automated Puppet Impact Analysis with Puppet Catalog Diff and GitLab CI
  • https://youtu.be/o8HP_wcxse4 - Puppet Camp Netherlands: Impact Analysis with Puppet Catalog Diff

Reference

Table of Contents

Classes

• catalog_diff::viewer: installs the catalog-diff viewer

Classes

catalog_diff::viewer

installs the catalog-diff viewer

Parameters

The following parameters are available in the catalog_diff::viewer class:

• remote
• password
• revision
• port
• listen_ip

remote

Data type: String

the url to the git repo of the catalog diff viewer

Default value: 'https://github.com/voxpupuli/puppet-catalog-diff-viewer.git'

password

Data type: String

password for basic authentication

Default value: 'puppet'

revision

Data type: String

the desired branch/rev that you want to checkout

Default value: 'master'

port

Data type: Integer

where apache will listen on

Default value: 1495

listen_ip

Data type: String

the desired ip adddress to listen on

Default value: $facts['networking']['ip']

Changelog

All notable changes to this project will be documented in this file. Each new release typically also includes the latest modulesync defaults. These should not affect the functionality of the module.

v4.1.0 (2024-10-03)

Full Changelog

Implemented enhancements:

• Allow trusted facts to be derived from node name #110 (nabertrand)
• reports: create them human readable #103 (bastelfreak)
• log destination for compiled catalogs #102 (bastelfreak)

Fixed bugs:

• Fix failed nodes report #108 (edward-harley)

Merged pull requests:

• Replace remaining invocations of obsolete URI.escape #123 (vchepkov)
• FacterDB: Switch to stringified facts #121 (bastelfreak)
• Delete unused spec_helper_local.rb #120 (bastelfreak)
• document all parameters with puppet-strings #119 (bastelfreak)
• Replace legacy fqdn fact with networking.fqdn #118 (bastelfreak)

v4.0.1 (2023-08-01)

Full Changelog

Fixed bugs:

• Include node stats in output report #94 (cyberkov)

Merged pull requests:

• Enhance README.md with more examples & fix typos #100 (bastelfreak)

v4.0.0 (2023-06-22)

Full Changelog

Breaking changes:

• Drop Puppet 6 support #98 (bastelfreak)

Implemented enhancements:

• Add puppet 8 support #97 (bastelfreak)

Merged pull requests:

• switch from camptocamp to voxpupuli module #96 (tuxmea)

v3.0.0 (2022-07-20)

Full Changelog

Breaking changes:

• Remove legacy Puppet manifest #36 (raphink)

Implemented enhancements:

• Document fact bootstrapping for new puppetserver #89 (bastelfreak)
• Support custom TLS certificates for old Puppetserver #87 (bastelfreak)
• Add support for custom certificates to old PuppetDB #72 (bastelfreak)
• Add Puppet 7 support #70 (bastelfreak)
• Implement support for custom PuppetDB URLs #68 (bastelfreak)
• comparer: exit early if checksum in old/new catalog is equal #65 (bastelfreak)
• compilecatalog: Switch to new HTTP client implementation #61 (bastelfreak)

Fixed bugs:

• comparer: exit earily if old/new contents are empty #66 (bastelfreak)
• Don't format Floats as Strings #54 (alexjfisher)

Closed issues:

• node_list doesn't work as intended #62
• Migrating this Repository to Voxpupuli #60
• doing diff on same node - 2 different environments? #41
• Saving a report doesn't work when passing two catalogs into the CLI #37

Merged pull requests:

• Document talks/blogposts about puppet-catalog-diff #86 (bastelfreak)
• Document --certless advantage/trusted facts setup #85 (bastelfreak)
• rubocop: various small cleanups part 4 #84 (bastelfreak)
• rubocop: various small cleanups part 3 #83 (bastelfreak)
• rubocop: various small cleanups part 1 #82 (bastelfreak)
• rubocop: various small cleanups part 1 #81 (bastelfreak)
• rubocop: Fix Style/TrailingCommaInArguments #80 (bastelfreak)
• rubocop: Fix Style/SymbolProc #79 (bastelfreak)
• rubocop: Fix Style/BlockDelimiters #78 (bastelfreak)
• rubocop: Fix Layout/IndentationWidth #77 (bastelfreak)
• rubocop: Fix Layout/EmptyLineAfterGuardClause #76 (bastelfreak)
• rubocop: Fix Layout/IndentationConsistency #75 (bastelfreak)
• rubocop: Fix Layout/EndAlignment #74 (bastelfreak)
• rubocop: Fix Layout/ElseAlignment #73 (bastelfreak)
• Clarify supported Puppet versions in README.md #71 (bastelfreak)
• Cleanup old code #67 (bastelfreak)
• comparer: cleanup trailing whitespace #64 (bastelfreak)
• auth.conf: Document how to update it with Puppet #63 (bastelfreak)
• Adding --node_list option #55 (serialh0bbyist)
• Make --output_report work when diffing two catalog files or directories. #50 (natemccurdy)
• Prevent 'false' from showing up in the console report #49 (natemccurdy)
• README.md: fix installation command #48 (aerickson)
• Fix threads option in pull face #40 (alexjfisher)
• Adds backward compatibility with older version of PuppetDB used in Puppet 3, to support Puppet 3 to 5 upgrades #39 (JohnEricson)
• Fix "Error: invalid byte sequence in UTF-8" error when retrieving catalogues with invalid encoding from PuppetDB on Puppet 3 server #38 (JohnEricson)

v2.3.0 (2020-08-18)

Full Changelog

Added

• Use prefer\_requested\_environment true #35 (alexjfisher)
• Include list of all\_changed\_nodes in json report #34 (alexjfisher)
• Hide sensitive parameters #33 (alexjfisher)
• Add link to upload_facts.rb script and documentation how to use it in Puppet upgrades to README #30 (JohnEricson)

v2.2.1 (2020-07-15)

Full Changelog

Fixed

• Get rid of obsolete URI.escape #29 (raphink)
• Fix problem where a node's details wouldn't load/show in puppet-catalog-diff-viewer #28 (JohnEricson)

v2.2.0 (2020-06-19)

Full Changelog

Added

• Add --exclude_resource_types= flag #27 (raphink)

v2.1.1 (2020-06-10)

Full Changelog

v2.1.0 (2020-06-10)

Full Changelog

Added

• Use #notice instead of #err for realtime message #26 (raphink)
• Add --exclude_defined_resources #25 (raphink)

v2.0.0 (2020-05-05)

Full Changelog

Changed

• Decommission old fact_search methods #17 (raphink)

Added

• Add coveralls #20 (raphink)
• Test comparer #18 (raphink)
• Update Readme #16 (raphink)
• Refactor/lint #15 (raphink)

Fixed

• Fix convert_pdb #21 (raphink)

v1.7.0 (2020-05-05)

Full Changelog

Added

• Refactor #14 (raphink)
• Lint #13 (raphink)
• Ignore parameters #12 (raphink)
• Sort hash keys in formater #11 (raphink)
• Include environment in report #10 (raphink)
• Update to current PDK template #9 (DavidS)
• Cache str diffs #8 (raphink)
• Puppetdb catalog #7 (raphink)
• Output endpoint in debug #6 (raphink)
• Support certless API #3 (raphink)
• Use v3 catalog API #2 (raphink)

Fixed

• Fix: v4/catalog returns an embedded catalog #5 (raphink)
• Check key in parsed catalog #1 (raphink)

v1.6.0 (2015-06-04)

Full Changelog

• The generation date (global info)
• Two node fields: old_version and new_version, using the catalog versions. #8 (raphink)

v1.5.2 (2015-06-03)

Full Changelog

• Improve puppetdb requests #7 (raphink)

v1.5.1 (2015-06-02)

Full Changelog

• Fix Typo #6 (claytono)

v1.5.0 (2015-06-02)

Full Changelog

• Multiple facts in filter #5 (raphink)
• Allow for filtering of local yaml cache #4 (raphink)
• Allow you to pass environment #2 (raphink)
• Update error handling #1 (supercow)
• Update puppedb option handling (raphink)

v1.0.0 (2015-06-02)

• Initial fork and release of RIAAs build (acidprime)

* This Changelog was automatically generated by github_changelog_generator