ptomulik-packagex_portsx

Table of Contents

0. Caution
1. Overview
2. Module Description
3. Setup
   • What portsx affects
   • Setup requirements
   • Beginning with packagex_portsx
4. Usage
5. Resolved issues
6. Known incompatibilities
7. Limitations
8. Development

Caution

This is an experimental module. It may be substantially changed, renamed or removed at all without a notice. Do not use in production.

Overview

NOTE: The build_options property is being renamed to package_settings. Currently it's a transition period so you may use one or the other. The build_options will be removed in next major release.

This is a portsx provider for packagex resource.

Module Description

The module re-implements puppet's ports provider adding some new features to it and fixing several existing issues. The new features include:

• install_options - extra CLI flags passed to portupgrade when installing, reinstalling and upgrading packages,
• uninstall_options - extra CLI flags passed to pkg_deinstall (old pkg toolstack) or pkg delete (pkgng) when uninstalling packages,
• package_settings - configuration options for package,
• build_options - an alias for package_settings, will be removed in next major release,
• works wit both the old pkg and new pkgng package databases,
• upgradeable (tested, the original puppet provider declared that it's upgradeable, but it never worked for me),
• portorigins (instead of portnames) are used to identify package instances,
• portversion is used to find installed packages (instead of pkg_info),
• make search is used to find uninstalled ports listed in puppet manifests,
• added unit tests (original provider had no tests),
• several issues resolved, see Resolved issues

The package_settings is simply an {OPTION => value} hash, with boolean values. The portsx provider ensures that package is compiled with prescribed package_settings. Normally you would set these options with make config command using ncurses-based frontend. Here, you can define package_settings in your puppet manifest. If a package is already installed and you change its package_settings in manifest file, the package gets rebuilt with new options and reinstalled.

Instead of portnames, portorigins are used to identify portsx instances (see FreeBSD ports collection and it's terminology). This copes with several problems caused by portnames' ambiguity (see FreeBSD ports collection and ambiguity of portnames). You can now install and mainain ports that have common portname (but different portorigins). Examples of such packages include mysql-client or ruby (see below).

The portversion utility is used to find installed ports. It's better than using pkg_info for several reasons. First, it is said to be faster, because it uses compiled version of ports INDEX file. Second, it works with both - the old pkg database and the new pkgng database, providing seamless interface to any of them. Third, it provides package names and their "out-of-date" statuses in a single call, so we don't need to separatelly check out-of-date status for installed packages. This version of portsx works with old pkg database as well as with pkgng, using portversion.

FreeBSD ports collection and its terminology

We use the following terminology when referring ports/packages:

• a string in form 'apache22' or 'ruby' is referred to as portname
• a string in form 'apache22-2.2.25' or 'ruby-1.8.7.371,1' is referred to as a pkgname
• a string in form 'www/apache22' or 'lang/ruby18' is referred to as a port origin or portorigin

See http://www.freebsd.org/doc/en/books/porters-handbook/makefile-naming.html

Port origins are used as primary identifiers for portsx instances. It's recommended to use portorigins instead of portnames as package names in manifest files.

FreeBSD ports collection and ambiguity of portnames

Using portnames (e.g. apache22) as package names in manifests is allowed. The portnames, however, are ambiguous, meaning that port search may find multiple ports matching the given portname. For example 'mysql-client' package has three ports at the time of this writing (2013-11-30): mysql-client-5.1.71, mysql-client-5.5.33, and mysql-client-5.6.13 with origins databases/mysql51-client, databases/mysql55-client and databases/mysql56-client respectively. If none of these ports are installed and you use this ambiguous portname in your manifest, you'll se the following warning:

Warning: Puppet::Type::Packagex::ProviderPortsx: Found 3 ports named 'mysql-client': 'databases/mysql51-client', 'databases/mysql55-client', 'databases/mysql56-client'. Only 'databases/mysql56-client' will be ensured.

Setup

What portsx affects

• installs, upgrades, reinstalls and uninstalls packages,
• modifies FreeBSD ports options' files /var/db/ports/*/options.local,

Setup Requirements

You may need to enable pluginsync in your puppet.conf.

Beginning with packagex_portsx

Its usage is essentially same as for the original ports provider. Here I just put some examples specific to new features.

Example 1 - using package settings

Using package_settings:

packagex { 'www/apache22': 
  package_settings => {'SUEXEC' => true}
}

Example 2 - using uninstall_options to cope with dependency problems

Sometimes freebsd package manager refuses to uninstall a package due to dependency problems that would appear after deinstallation. In such situations we may use the uninstall_options to instruct the provider to uninstall also all packages that depend on the package being uninstalled. When using ports with old pkg package manager one would write in its manifest:

packagex { 'www/apache22':
  ensure => absent,
  uninstall_options => ['-r'] 
}

For pkgng one has to write:

packagex { 'www/apache22':
  ensure => absent,
  uninstall_options => ['-R','-y'] 
}

Example 3 - using install_options

The new portsx provider implements install_options feature. The flags provided via install_options are passed to portupgrade command when installing, reinstalling or upgrading packages. With no install_options provided, sensible defaults are selected by portsx provider.

Let's say we want to install precompiled package, if available (-P flag). Write the following manifest:

packagex { 'www/apache22':
  ensure => present,
  install_options => ['-P', '-M', {'BATCH' => 'yes'}]
}

Now, if we run puppet, we'll see the command:

~ # puppet agent -t --debug --trace
...
Debug: Executing '/usr/local/sbin/portupgrade -N -P -M BATCH=yes www/apache22'
...

Note, that the portsx provider adds some flags by its own (-N in the above example). What is added/removed is preciselly stated in provider's generated documentation.

Usage

I think, there is nothing worth to be written in addition to Beginning with portsx.

Resolved issues

Outdated ports get installed when portorigin is used in site.pp

The test case is following (2013.11.30):

• package mysql-client is absent initially,
• there are three ports available in ports tree that share mysql-client portname :
  • databases/mysql51-client (oldest),
  • databases/mysql55-client, and
  • databases/mysql55-client (most recent),
• the site.pp contains: package {'mysql-client': ensure => present} or package {'mysql-client': ensure => latest},

Note, that the situation with mysql-client has changed recently such that databases/mysql55-client uses mysql55-client as portname for example, but it was the case a little bit earlier that mysql-client was used by all three ports. Shared portnames are still used by some other packages, however.

~ # puppet agent -t --debug --trace
...
Debug: Executing '/usr/local/sbin/portupgrade -N -M BATCH=yes mysql-client'
...

after installation we have:

~ # portversion -v -o mysql-client
databases/mysql51-client    =  up-to-date with port

which is certainly not the most recent available version.

The same case, but with new packagex provider yields:

~ # puppet agent -t --debug --trace
...
Warning: Puppet::Type::Packagex::ProviderPortsx: Found 3 ports named 'mysql-client': 'databases/mysql51-client', 'databases/mysql55-client', 'databases/mysql56-client'. Only 'databases/mysql56-client' will be ensured.
Debug: Executing '/usr/local/sbin/portupgrade -N -M BATCH=yes databases/mysql56-client'
...

on output and afer installation we have:

~ # portversion -v -o mysql-client
databases/mysql56-client    =  up-to-date with port

The reason for the old ports provider to pickup outdated port is the following. If we manually run the portupgrade command, we'll see:

~ # /usr/local/sbin/portupgrade -N -M BATCH=yes mysql-client
--->  Found 3 ports matching 'mysql-client':
        databases/mysql51-client
        databases/mysql55-client
        databases/mysql56-client
Install 'databases/mysql51-client'? [yes]

The old ports provider simply says y here and installs first proposed port.

Package resources are not properly listed with puppet resource package command

The test case is following (2013.12.13):

• several ports with portname docbook.* are installed at the same time,

Running portversion yields:

~ # portversion -Q -o 2>/dev/null | grep docbook
textproc/docbook
textproc/docbook-410
textproc/docbook-420
textproc/docbook-430
textproc/docbook-440
textproc/docbook-450
textproc/docbook-500
textproc/docbook-sk
textproc/docbook-xml
textproc/docbook-xml-430
textproc/docbook-xml-440
textproc/docbook-xml-450
textproc/docbook-xsl

whereas running puppet resource package:

~ # puppet resource package | grep '^package {' | grep  docbook
package { 'docbook':
package { 'docbook-sk':
package { 'docbook-xml':
package { 'docbook-xsl':

The same case, but with the new portsx provider yields:

~ # puppet resource packagex | grep '^packagex {' | grep  docbook
packagex { 'textproc/docbook':
packagex { 'textproc/docbook-410':
packagex { 'textproc/docbook-420':
packagex { 'textproc/docbook-430':
packagex { 'textproc/docbook-440':
packagex { 'textproc/docbook-450':
packagex { 'textproc/docbook-500':
packagex { 'textproc/docbook-sk':
packagex { 'textproc/docbook-xml':
packagex { 'textproc/docbook-xml-430':
packagex { 'textproc/docbook-xml-440':
packagex { 'textproc/docbook-xml-450':
packagex { 'textproc/docbook-xsl':

that is the information from the new portsx provider agrees with that obtained from package manager.

With ensure => latest packages are not upgraded

The test case is following (2013.12.1):

• help2man-1.43.3 is initially installed,
• new version help2man-1.43.3_1 is available in ports tree:

~ # portversion -v help2man
help2man-1.43.3             <  needs updating (port has 1.43.3_1)

• we have site.pp with package { 'help2man': ensure => latest}

If we now run puppet, we'll see:

~ # puppet agent -t --debug --trace
...
Debug: /Stage[main]//Node[puppet-test.mgmt.meil.pw.edu.pl]/Package[help2man]/ensure: help2man "1.43.3" is installed, latest is "1.43.3_1"
Debug: Executing '/usr/local/sbin/portupgrade -N -M BATCH=yes help2man'
Notice: /Stage[main]//Node[puppet-test.mgmt.meil.pw.edu.pl]/Package[help2man]/ensure: ensure changed '1.43.3' to '1.43.3_1'
...
...

However, after that the outated version of package is still installed:

~ # portversion -v help2man
help2man-1.43.3             <  needs updating (port has 1.43.3_1)

The reason becames obvious, when we run the portupgrade command manually:

~ # /usr/local/sbin/portupgrade -N -M BATCH=yes help2man
** Found already installed package(s) of 'misc/help2man': help2man-1.43.3

The -N flag shouldn't be here. Correct command line is:

~ # /usr/local/sbin/portupgrade -R -M BATCH=yes help2man

and is used by the new portsx provider. If we use the new portsx provider, the package upgrades smoothly:

~ # puppet agent -t --debug --trace
...
Debug: Packagex[help2man](provider=portsx): Newer version in port
Debug: /Stage[main]//Node[puppet-test.mgmt.meil.pw.edu.pl]/Packagex[help2man]/ensure: help2man "1.43.3" is installed, latest is "1.43.3_1"
Debug: Executing '/usr/local/sbin/portupgrade -R -M BATCH=yes misc/help2man'
...

and after that:

~ # portversion -v help2man
help2man-1.43.3_1           =  up-to-date with port

Note, for testing purposes, you may downgrade package with portdowngrade tool.

The package upgrade fails if portname changes between versions

The test case is (2013.12.1):

• installed is mysql-client-5.5.31 (databases/mysl55-client)
• the new version is available:

~ # portversion -v mysql-client
mysql-client-5.5.31         <  needs updating (port has 5.5.34)

• the portname changes from mysql-client to mysql55-client between versions.
• the site.pp contains: package{'mysql-client': ensure => latest}

If we run puppet, it fails to upgrade the port:

~ # puppet agent -t --debug --trace
...
Debug: Executing '/usr/local/sbin/portupgrade -N -M BATCH=yes mysql-client'
Error: Could not update: Could not find package mysql-client
/usr/local/lib/ruby/site_ruby/1.9/puppet/util/errors.rb:96:in `fail'
/usr/local/lib/ruby/site_ruby/1.9/puppet/type/package.rb:93:in `rescue in block (3 levels) in <module:Puppet>'
/usr/local/lib/ruby/site_ruby/1.9/puppet/type/package.rb:90:in `block (3 levels) in <module:Puppet>'</module:Puppet></module:Puppet>
...

Running the portupgrade command manually reveals the main problem:

~ # /usr/local/sbin/portupgrade -N -M BATCH=yes mysql-client
** No such package or port: mysql-client

The new portsx provider operates on portorigins and the upgrade runs without problem:

~ # puppet agent -t --debug --trace
...
Debug: Packagex[mysql-client](provider=portsx): Newer version in port
Debug: /Stage[main]//Node[puppet-test.mgmt.meil.pw.edu.pl]/Packagex[mysql-client]/ensure: mysql-client "5.5.31" is installed, latest is "5.5.34"
Debug: Executing '/usr/local/sbin/portupgrade -R -M BATCH=yes databases/mysql55-client'
...

Note, that the new portsx provider will fail at next transaction (the one after successful upgrade). This is because the name mysql-clients disappeared from both the ports and packages database. The manifest file must be updated to reflect change from mysql-client to mysql55-client. The better option through would be to use portorigin databases/mysql55-client in site.pp. Note, that this helps only for the new portsx provider (the old still doesn't work due to the -N flag issue).

Uninstall fails when there are other packages that depend on this one

The tests case is (2013.12.1):

• package apache22-event-mpm-2.2.25 is initially installed,
• other packages that depend on it are initially installed, for example:
  • ap22-mod_rpaf2-0.6_3
  • portdowngrade-1.4
  • subversion-1.8.3
• the site.pp contains package{'www/apache22-event-mpm': ensure => absent}

If we invoke puppet, it runs into trouble:

~ # puppet agent -t --debug --trace
...
Debug: Executing '/usr/local/sbin/pkg_deinstall www/apache22-event-mpm'
Error: Execution of '/usr/local/sbin/pkg_deinstall www/apache22-event-mpm' returned 1: --->  Deinstalling 'apache22-event-mpm-2.2.25'
pkg_delete: package 'apache22-event-mpm-2.2.25' is required by these other packages
and may not be deinstalled:
ap22-mod_rpaf2-0.6_3
portdowngrade-1.4
subversion-1.8.3
** Listing the failed packages (-:ignored / *:skipped / !:failed)
        ! apache22-event-mpm-2.2.25     (pkg_delete failed)
...

Now, if we use the new portsx provider with appropriate uninstall_options, it is again able to uninstall the package (and all the other packages that depend on it if needed). For example, one may use -r flag if the old pkg toolstack is used to manage packages:

packagex { 'www/apache22-event-mpm':
           ensure => absent,
           uninstall_options => ['-r'] }

Then running puppet, we can easilly uninstall the package(s) recursivelly.:

~ # puppet agent -t --debug --trace
...
Debug: Executing '/usr/local/sbin/pkg_deinstall -r apache22-event-mpm-2.2.25'
Notice: /Stage[main]//Node[puppet-test.mgmt.meil.pw.edu.pl]/Packagex[www/apache22-event-mpm]/ensure: removed
...
Notice: Finished catalog run in 15.09 seconds

Known incompatibilities

Some design decisions caused, that there are incompatibilities w.r.t original version of ports provider

Portorigins used internally to identify packages.

This may cause some troubles to scrips/manifests that depend on package names held by the portsx provider. If there are some scripts, for example, which parse the output of puppet resource package ..., then they may fail with new portsx provider. To exemplify this, let's see the output of puppet resource ... for the old ports provider and new portsx provider.

For the old one we have, for example:

~ # puppet resource package | grep 'package {' | grep "autoconf':"
package { 'autoconf':

whereas the same for the new implementation is:

~ # puppet resource packagex | grep 'packagex {' | grep "autoconf':"
packagex { 'devel/autoconf':

See the difference in the package name?

The puppet resource packagex displays package_settings

This too may break some scripts that parse output of puppet resource packagex .... The example output for package having package_settings is:

~ # puppet resource packagex 'textproc/libxml2'
packagex { 'textproc/libxml2':
  ensure        => '2.8.0_3',
    package_settings => '{:MEM_DEBUG=>false, :SCHEMA=>true, :THREADS=>true, :THREAD_ALLOC=>false, :XMLLINT_HIST=>false}',
    }

Note, that the package_settings would never appear in output of the original (old) ports provider.

Limitations

• If there are several ports installed with same portname - for example docbook - then puppet resource packagex docbook will list only one of them (the last one from portversions list - usually the most recent). It is so, because portsx uses portorigins to identify its instances (as name paramateter). None of the existing instances is identified by puppet as an instance of docbook and puppet falls back to use provider's query method. But query handles only one package per name (in this case the last one from portversion's list if chosen). This is an issue, which will not probably be fixed, so you're encouraged to use portorigins.
• Currently there is no system tests for the new portsx provider. This is, because there are no FreeBSD prefab images provided by rspec-system yet. I hope this changes in not so far future, see status of the request for freebsd prefab images.

Development

The project is held at github:

• https://github.com/ptomulik/puppet-packagex Issue reports, patches, pull requests are welcome!

2014-01-19 Pawel Tomulik ptomulik@meil.pw.edu.pl

• release 0.1.6
• fixed buggy command line string for pkgng uninstall action 2014-01-18 Pawel Tomulik ptomulik@meil.pw.edu.pl
• fixed LICENSE
• corrections to README.md
• release 0.1.5
• fixed problem with loading modules from 'portsxutil' 2014-01-17 Pawel Tomulik ptomulik@meil.pw.edu.pl
• release 0.1.4
• updated README.md to notice about package_settings stuff 2014-01-10 Pawel Tomulik ptomulik@meil.pw.edu.pl
• release 0.1.3
• added portsxutil to $LOAD_PATH in provider 2014-01-09 Pawel Tomulik ptomulik@meil.pw.edu.pl
• release 0.1.2
• notified about transition from build_options to package_settings 2014-01-07 Pawel Tomulik ptomulik@meil.pw.edu.pl
• removed build_options from resource docs
• changed name for build_options to package_settings 2014-01-06 Pawel Tomulik ptomulik@meil.pw.edu.pl
• removed codeclimage badge from README.md
• release 0.1.1
• fixed names in .scripts/build-module.sh
• fixed dependencies in Modulefile
• release 0.1.0
• added .scripts/
• initial commit