github_inventory

Table of Contents

• Description
  • Why is this useful?
  • How is this useful with Bolt?
• Setup
  • Setup Requirements
  • Beginning with github_inventory
• Usage
  • Using the plugin in a Bolt inventory file
• Reference
• Limitations
  • Targets' .facts keys use _name instead of name
  • Limitations when running the example project
• Development

Description

github_inventory is an inventory reference plugin for Puppet Bolt. It uses the GitHub API to dynamically provide a collection of local transport Targets that represent each repository under a GitHub organization or user.

This module also contains an example Bolt project with a working inventory.yaml and several Bolt plans.

Why is this useful?

Modeling GitHub repositories as Inventory Targets lets admins orchestrate org-wide repo operations at scale, using simple & reusable Bolt Plans.

How is this useful with Bolt?

Simplicity, speed, and reusability: It's trivial to define an inventory, and Bolt plans are generally concise and legible. And when using multiple Targets (repos) with Bolt plan functions like run_*()/parallelize(), you get concurrent execution built-in for free. Plans are easy to wrap up in a module and share, so the next time someone has to reset all the required PR checks in a 200-repo org, they can just pull out the plan and run it on targets from their org.

Each github_inventory Target describes its repository in its .facts, using the same keys as the data structure returned by GitHub's /orgs/{org}/repos or /repos/{username}/repos endpoints (with one important exception).

Targets use the local transport by default. This keeps execution as frictionless as possible and ensures that a known version of Ruby (the Bolt interpreter's) is available to execute tasks. It also also opens the door to advanced uses, like git-cloning repos en masse into local working directories and enforcing file conventions with Tasks and apply() blocks.

Note: Targets don't use the remote Transport. It can only run remote Tasks (which rules out the built-in http_request) and can't compile Puppet apply() blocks.

Setup

Setup Requirements

• Puppet Bolt 3.0+ or 2.37, installed from an OS package (don't use the RubyGem)
• A GitHub API personal auth token with sufficient scope
• The octokit RubyGem

Beginning with github_inventory

1. If you are using rvm, you must disable it before running bolt:

   rvm use system
   

2. Install the RubyGem dependencies using Bolt's gem command

   On most platforms:

   /opt/puppetlabs/bolt/bin/gem install --user-install -g gem.deps.rb
   

   On Windows:

   "C:/Program Files/Puppet Labs/Bolt/bin/gem.bat" install --user-install -g gem.deps.rb
   

3. (If using the example Bolt plans in this module)

   Set the environment variables GITHUB_ORG and GITHUB_API_TOKEN to appropriate values for your GitHub organization. The API token needs a scope that can query the organization's repos.

Usage

To use this plugin in your own Bolt project, configure it to provide targets in the inventory file.

Using the plugin in a Bolt inventory file

An example inventory.yaml file:

groups:
  - name: repo_targets
    targets:
      - _plugin: github_inventory  # <- Plugin provides `local` Targets
        org: simp                  # <- GitHub org with Target repos
        github_api_token:          # <- API token with scope that can get repos
          _plugin: env_var         # <- (provided by another Bolt plugin)
          var: GITHUB_API_TOKEN

config:
  transport: local
  local:
    interpreters:
      .rb: /opt/puppetlabs/bolt/bin/ruby
    tmpdir:
     _plugin: env_var
     var: PWD

Reference

See REFERENCE.md

Limitations

Targets' .facts keys use _name instead of name

• github_inventory Target facts use the key _name instead of the repo-level name key.
• This avoids compilation errors in Puppet apply() blocks.

Limitations when running the example project

In order to provide the example bolt project in the same module as the inventory plugin, modules/github_inventory/ is symlinked to the repo's top-level directory. This allows the bolt project to find the inventory plugin, but there are some side-effects:

• Bolt will emit a (benign) warning near the beginning of each run:

  The project 'github_inventory' shadows an existing module of the same name [ID: project_shadows_module]
  

• Git may not establish the symlink properly for Windows users

This quirk only affects the example bolt project; it will not affect the inventory plugin, or running the github_inventory:: plans from your own Bolt project.

Development

Submit PRs on the project's GitHub page.

Reference

Table of Contents

Tasks

• resolve_reference: Return a GitHub organization's repositories as local inventory targets

Plans

• github_inventory::clone_git_repos: Clone all repos into a local directory
• github_inventory::count: Example plan, prints number of Targets from inventory
• github_inventory::latest_semver_tags: Report the highest SemVer tag for each repo (that has SemVer tags), including information release (if a release exists for that tag) and uplo
• github_inventory::required_checks: List and/or set which PR checks are required on each repo
• github_inventory::update_forked_mirrors: Update default branches & tags on forked GitHub repos (with an option to only affect Puppet module projects for a specific forge org) For ea
• github_inventory::workflows: Return repos with GitHub Actions workflows

Tasks

resolve_reference

Return a GitHub organization's repositories as local inventory targets

Supports noop? false

Parameters

org

Data type: String[1]

GitHub org name (or user login) with repos

github_api_token

Data type: Optional[String[1]]

Optional GitHub personal OAuth token, which may be useful to avoid the GitHub API's unauthenticated rate limits

archived_repos

Data type: Boolean

When true, includes archived repositories in results.

private_repos

Data type: Boolean

When true, includes private repositories in results.

allow_list

Data type: Optional[Array[String[1]]]

repo names/patterns to include in inventory, drops all other repos

block_list

Data type: Optional[Array[String[1]]]

repo names/patterns to reject from inventory (can reject targets in allow_list)

transport_type

Data type: String[1]

Bolt Transport type of repository Targets

accept_header

Data type: String[1]

Override to request a custom media type from the GitHub API

extra_gem_path

Data type: Optional[String[1]]

Additional GEM_PATH path for ruby gems (to find octokit)

Plans

github_inventory::clone_git_repos

Clone all repos into a local directory

Parameters

The following parameters are available in the github_inventory::clone_git_repos plan:

• targets
• target_dir
• collision_strategy
• clone_protocol
• return_result

targets

Data type: TargetSpec

Name of github_inventory Targets (or inventory group)

Default value: 'github_repos'

target_dir

Data type: Stdlib::Absolutepath

Local directory to clone repos into

Default value: "${system::env('PWD')}/_repos"

collision_strategy

Data type: Enum[fail,skip,overwrite,fetch]

Action to take when a local repo directory already exists

Default value: 'skip'

clone_protocol

Data type: Enum[http,ssh]

'http' or 'ssh'

Default value: 'http'

return_result

Data type: Boolean

When true, plan returns data in a ResultSet

Default value: false

github_inventory::count

Example plan, prints number of Targets from inventory

Parameters

The following parameters are available in the github_inventory::count plan:

• targets
• display_result
• return_result

targets

Data type: TargetSpec

Name of github_inventory Targets (or inventory group)

Default value: 'github_repos'

display_result

Data type: Boolean

When true, plan prints result using out::message

Default value: true

return_result

Data type: Boolean

When true, plan returns data in a ResultSet

Default value: false

github_inventory::latest_semver_tags

Report the highest SemVer tag for each repo (that has SemVer tags), including information release (if a release exists for that tag) and uploaded assets

• Note reports repos with "SemVer-ish" tags (includes /^v/ and /-d$/)

Parameters

The following parameters are available in the github_inventory::latest_semver_tags plan:

• targets
• github_api_token
• display_result
• return_result

targets

Data type: TargetSpec

Name of github_inventory Targets (or inventory group)

Default value: 'github_repos'

github_api_token

Data type: Sensitive[String[1]]

GitHub API token. Doesn't require any scope for public repos.

Default value: (system::env('GITHUB_API_TOKEN'))

display_result

Data type: Boolean

When true, plan prints result using out::message

Default value: true

return_result

Data type: Boolean

When true, plan returns data in a ResultSet

Default value: false

github_inventory::required_checks

List and/or set which PR checks are required on each repo

Parameters

The following parameters are available in the github_inventory::required_checks plan:

• targets
• github_api_token
• checks

targets

Data type: TargetSpec

Name of github_inventory Targets (or inventory group)

Default value: 'github_repos'

github_api_token

Data type: Sensitive[String[1]]

GitHub API token. Needs repo scope to see or set checks.

Default value: (system::env('GITHUB_API_TOKEN'))

checks

Data type: Optional[String[1]]

Optional comma-delimited list of required PR Checks to set on all repos If defined, this will overwrite ALL repos' required PR checks

Default value: undef

github_inventory::update_forked_mirrors

Update default branches & tags on forked GitHub repos (with an option to only affect Puppet module projects for a specific forge org)

For each target repo (provided by github_inventory plugin):

• Clone repo
• (Optional) skip if repo is not a Puppet module for desired forge_org
• Add parent repo as remote, fetchparent's default branch and tags
• (Optional) skip if noop or repo is in noop_repos
• Push parent's default branch to repo (and push tags on branch)
• Ensure repo's default branch matches parent's default branch

Target repos can be fine-tuned in the inventory by using the allow_list/block_list parameters in the github_inventory plugin

Parameters

The following parameters are available in the github_inventory::update_forked_mirrors plan:

• targets
• target_dir
• noop
• noop_repos
• github_api_token
• clone_repos
• forge_org

targets

Data type: TargetSpec

Name of github_inventory Targets (or inventory group)

Default value: 'github_repos'

target_dir

Data type: Stdlib::Absolutepath

Local directory to clone repos into (when clone_repos = true)

Default value: "${system::env('PWD')}/_repos"

noop

Data type: Boolean

When true, all repos will run through all prep steps, but not push up changes.

Default value: true

noop_repos

Data type: Array[String,0]

List of specific repos to always treat as noop, even when noop=false

Default value: []

github_api_token

Data type: Sensitive[String[1]]

Default value: (system::env('GITHUB_API_TOKEN'))

clone_repos

Data type: Boolean

Default value: true

forge_org

Data type: Optional[String[1]]

Default value: 'puppetlabs'

github_inventory::workflows

Return repos with GitHub Actions workflows

Parameters

The following parameters are available in the github_inventory::workflows plan:

• targets
• github_api_token

targets

Data type: TargetSpec

By default: github_repos group from inventory

Default value: 'github_repos'

github_api_token

Data type: Sensitive[String[1]]

GitHub API token. Doesn't require any scope for public repos.

Default value: (system::env('GITHUB_API_TOKEN'))

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog.

0.4.2 - 2021-07-19

Fixed

• Slimmed down overwhelming amount of inconsequential user data returned by example plan github_inventory::latest_semver_tags

0.4.1 - 2021-07-19

Fixed

• Fixed version comparison bug in example plan github_inventory::latest_semver_tags

0.4.0

Added

• New example plan github_inventory::update_forked_mirrors, which keeps a forked repos' default branch & tags synced to its parent repo
• Add associated release data (if present) for tags returned by github_inventory::latest_semver_tags
• Use forge-based puppet-debug module

0.3.0

Added

• New parameter accept_header in inventory plugin
• New parameters collision_strategy, clone_protocol, return_result in example plan github_inventory::clone_git_repos
• New parameters display_result and return_result in the example plan github_inventory::latest_semver_tags

Fixed

• Fixed misnamed variable in example plan github_inventory::clone_git_repos

0.2.1

Fixed

• Fixed bug that caused String $targets to fail in the example plan github_inventory::workflows (by adding get_targets())

Changed

• Renamed example inventory.yaml group to github_repos to play more nicely with other projects' inventory.yaml

0.2.0

Added

• New example plan, github_inventory::git_clone, which clones repo Targets into a local directory.

Changed

• Renamed Target fact name to _name to prevent Bolt apply() Puppet compiles from failing with Cannot reassign variable '$name' errors

Fixed

• Fixed syntax and logic bugs in example plan github_inventory::count

0.1.0

Added

• Initial project
• Inventory plugin that returns GitHub org repos as local transport Targets
• Example Bolt project with working Plans and inventory.yaml