Forge Home

Modules

Puppet

Resources

PuppetEcosystemPuppet CommunityUpdates

hiera_vault

Hiera 5 backend to query data lookups from vault
Project URLRSS FeedReport issues

174,371 downloads

32,123 latest version

5.0 quality score

We run a couple of automated
scans to help you access a
module's quality. Each module is
given a score based on how well
the author has formatted their
code and documentation and
modules are also checked for
malware using VirusTotal.

Please note, the information below
is for guidance only and neither of
these methods should be considered
an endorsement by Puppet.

Version information

  • 2.0.0 (latest)
  • 1.0.0
  • 0.4.0
  • 0.3.0
  • 0.2.0
  • 0.1.0
released Jul 29th 2021
This version is compatible with:
  • Puppet Enterprise 2019.8.x, 2019.7.x, 2019.5.x, 2019.4.x, 2019.3.x, 2019.2.x, 2019.1.x, 2019.0.x, 2018.1.x, 2017.3.x, 2017.2.x, 2016.4.x
  • Puppet >= 4.10.0 < 7.0.0
  • , ,

Start using this module

  • r10k or Code Manager
  • Bolt
  • Manual installation
  • Direct download

Add this module to your Puppetfile:

mod 'petems-hiera_vault', '2.0.0'
Learn more about managing modules with a Puppetfile

Add this module to your Bolt project:

bolt module add petems-hiera_vault
Learn more about using this module with an existing project

Manually install this module globally with Puppet module tool:

puppet module install petems-hiera_vault --version 2.0.0

Direct download is not typically how you would use a Puppet module to manage your infrastructure, but you may want to download the module in order to inspect the code.

Download

Documentation

petems/hiera_vault — version 2.0.0 Jul 29th 2021

hiera_vault : a vault data provider function (backend) for Hiera 5

Description

Warning: master may be broken whilst this repo is upgraded for k/v v2 and newer Vault version upgrades! Please use the 0.1.0 tagged release in the meantime. This message will be removed and a 1.0.0 breaking release will be tagged on the Forge in the future.

This is a back end function for Hiera 5 that allows lookup to be sourced from Hashicorp's Vault.

Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. Vault handles leasing, key revocation, key rolling, and auditing. Vault presents a unified API to access multiple backends: HSMs, AWS IAM, SQL databases, raw key/value, and more.

For an example repo of it in action, check out the hashicorp/webinar-vault-hiera-puppet repo and webinar 'How to Use HashiCorp Vault with Hiera 5 for Secret Management with Puppet'

Compatibility

  • This module is only compatible with Hiera 5 (ships with Puppet 4.9+) and Vault KV engine version 2 (Vault 0.10+)

Requirements

The vault and debouncer gems must be installed and loadable from Puppet

# /opt/puppetlabs/puppet/bin/gem install --user-install vault
# /opt/puppetlabs/puppet/bin/gem install --user-install debouncer
# puppetserver gem install vault
# puppetserver gem install debouncer

On Puppetserver <= 5, you will need to switch Puppetserver to use the new JRuby 9K, as the gem requires Ruby 2+, and Puppetserver uses the 1.9.2 JRuby

Some example Puppetcode to do so:

ini_setting { "Change jruby to 9k":
  ensure            => present,
  setting           => 'JRUBY_JAR',
  path              => "/etc/sysconfig/puppetserver",
  key_val_separator => '=',
  section           => '',
  value             => '"/opt/puppetlabs/server/apps/puppetserver/jruby-9k.jar"',
  show_diff         => true,
  notify            => Service['puppetserver']
}

package { 'vault-puppetserver-gem':
  ensure   => 'present',
  name     => 'vault',
  provider => 'puppetserver_gem',
}
->
package { 'vault-puppetpath-gem':
  ensure   => 'present',
  name     => 'vault',
  provider => 'puppet_gem',
}
->
package { 'debouncer-puppetserver-gem':
  ensure   => 'present',
  name     => 'debouncer',
  provider => 'puppetserver_gem',
}
->
package { 'debouncer-puppetpath-gem':
  ensure   => 'present',
  name     => 'debouncer',
  provider => 'puppet_gem',
}
~> Service['puppetserver']

On Puppetserver >= 6, this is not needed as the default has been moved to the newer JRuby.

Installation

The data provider is available by installing the petems/hiera_vault module into your environment:

This will avaliable on the forge, and installable with the module command:

# puppet module install petems/hiera_vault

You can also download the module directly:

git clone https://github.com/petems/petems-hiera_vault /etc/puppetlabs/code/environments/production/modules/hiera_vault

Or add it to your Puppetfile

mod 'hiera_vault',
  :git => 'https://github.com/petems/petems-hiera_vault'

Hiera Configuration

See The official Puppet documentation for more details on configuring Hiera 5.

The following is an example Hiera 5 hiera.yaml configuration for use with hiera-vault

---
version: 5

hierarchy:
  - name: "Hiera-vault lookup"
    lookup_key: hiera_vault
    options:
      confine_to_keys:
        - "^vault_.*"
        - "^.*_password$"
        - "^password.*"
      ssl_verify: false
      address: https://vault.foobar.com:8200
      token: <insert-your-vault-token-here>
      default_field: value
      mounts:
        some_secret:
          - %{::trusted.certname}
          - common
        another_secret:
          - %{::trusted.certname}
          - common

The following mandatory Hiera 5 options must be set for each level of the hierarchy.

name: A human readable name for the lookup

lookup_key: This option must be set to hiera_vault

The following are optional configuration parameters supported in the options hash of the Hiera 5 config

address: The address of the Vault server or Vault Agent, also read as ENV["VAULT_ADDR"]. Note: Not currently compatible with unix domain sockets - you must use http:// or https://

token: The token to authenticate with Vault, also read as ENV["VAULT_TOKEN"] or a full path to the file with the token (eg. /etc/vault_token.txt). When bootstrapping, you can set this token as IGNORE-VAULT and the backend will be stubbed, which can be useful when bootstrapping.

confine_to_keys:: Only use this backend if the key matches one of the regexes in the array, to avoid constantly reaching out to Vault for every parameter lookup

confine_to_keys:
  - "application.*"
  - "apache::.*"

strip_from_keys: Patterns to strip from keys before lookup

strip_from_keys:
  - "vault:"

default_field:: The default field within data to return. If not present, the lookup will be the full contents of the secret data.

mounts:: The list of mounts you want to do lookups against. This is treated as the backend hiearchy for lookup. It is recomended you use Trusted Facts within the hierachy to ensure lookups are restricted to the correct hierachy points. See Mounts.

:ssl_verify: Specify whether to verify SSL certificates (default: true)

v1_lookup: whether to lookup within kv v1 hierarchy (default true) - disable if you only use kv v2 :) See Less lookups.

v2_guess_mount: whether to try to guess mount for KV v2 (default true) - add data after your mount and disable to minimize amount of misses. See Less lookups.

Debugging

puppet lookup vault_notify --explain --compile --node=node1.vm
Searching for "vault_notify"
  Global Data Provider (hiera configuration version 3)
    Using configuration "/etc/puppetlabs/code/hiera.yaml"
    Hierarchy entry "yaml"
      Path "/etc/puppetlabs/code/environments/production/hieradata/node1.yaml"
        Original path: "%{::hostname}"
        No such key: "vault_notify"
      Path "/etc/puppetlabs/code/environments/production/hieradata/common.yaml"
        Original path: "common"
        Path not found
  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
    Hierarchy entry "Hiera-vault lookup"
      Found key: "vault_notify" value: "hello123"

Vault Configuration

Mounts

It is recomended to have a specific mount for your Puppet secrets, to avoid conflicts with an existing secrets backend.

From the command line:

vault secrets enable -version=2 -path=some_secret kv

We will then configure this in our hiera config:

mounts:
  some_secret:
    - %{::trusted.certname}
    - common

Then when a hiera call is made with lookup on a machine with the certname of foo.example.com:

$cool_key = lookup({"name" => "cool_key", "default_value" => "No Vault Secret Found"})

Secrets will then be looked up with the following paths:

Less lookups

You can use v1_lookup and v2_guess_mount to minimize misses in above lookups.

Changing above configuration to

v2_guess_mount: false
v1_lookup: false
mounts:
  some_secret/data:
    - %{::trusted.certname}
    - common

would result in following lookups:

Multiple keys in trusted certname

Often you want to whitelist multiple paths for each host (e.g. due to host having multiple roles). In this case simply add keys delimited with comma to trusted field. For example:

mounts:
  secret:
    - "%{trusted.extensions.pp_role}"

and host configured with

---
extension_requests:
  pp_role: api,ssl

would result in lookups in:

More verbose paths in Hiera

Often implicit path extension makes it hard to understand which exact paths are used for given host - as you need to inspect both Hiera and trusted field for each host.

With above configuration and lookup $cool_key = lookup({"name" => "cool_key"}) you cannot be sure whether api/cool_key or ssl/cool_key will be used (whichever happens to be first in lookup list).

To alleviate this problem you can use full paths in Hiera, provided v2_guess_mount: false configuration is active. For example with:

v2_guess_mount: false
v1_lookup: false
mounts:
  secret/data:
    - "%{trusted.extensions.pp_role}"

You can use $cool_key = lookup({"name" => "ssl/cool_key"}) to ensure http://vault.foobar.com:8200/secret/data/ssl/cool_key will be used.

And make yourself a favor and avoid lookup directly ;) Use

profile::ssl_role::key: "%{alias('vault_storage::ssl/params.key')}"

to inject value from key inside http://vault.foobar.com:8200/secret/data/ssl/params.

Author

  • Original - David Alden dave@alden.name
  • Transfered and maintained by Peter Souter