Masterfiles Policy Framework

Table of contents

The Masterfiles Policy Framework or MPF also commonly referred to as simply masterfiles is the policy framework that ships with CFEngine.

The Masterfiles Policy Framework (MPF) is the default policy that ships with both the CFEngine Enterprise and Community editions. The MPF includes policy to manage CFEngine itself, the standard library (stdlib), and policy to inventory various aspects of the system.

Overview

update.cf - The update policy entry.
promises.cf - The main policy entry.
lib/ - The standard library.
inventory/ - Inventory policy.
services - User defined custom policy.
services/main.cf - Contains an empty bundle agent main where custom policy can be integrated.
services/autorun/ - Automatically included policy files.
.no-distrib/ - A directory that is excluded from policy updates from remote agents.

The MPF is continually updated. You can track its development on github.

Configuration

The most common controls reference variables in the def bundle in order to keep the modifications to the distributed policy contained within a single place. The recommended way to set classes and define variables in the def bundle is using an augments file. Keeping the modifications to the distributed policy set makes policy framework upgrades significantly easier.

Note: If you need to make modification to a shipped file consider opening a pull request to expose the tunable into the def bundle.

Note: controls/def.cf contains the defaults and settings for promises.cf and controls/update_def.cf contains the defaults and settings for update.cf.

History:

In 3.7.8, 3.10.4, and 3.12.0 the class cf_runagent_initiated is defined by default in the MPF for agent executions initiated by cf-runagent through cf-serverd. Previously the class cfruncommand was defined. See body server control cfruncommand in controls/cf_serverd.cf.

Update policy (update.cf)

Synchronizing clients with the policy server happens here, in update.cf. Its main job is to copy all the files on the policy server (usually the hub) under $(sys.masterdir) (usually /var/cfengine/masterfiles) to the local host into $(sys.inputdir) (usually /var/cfengine/inputs).

This file should rarely if ever change. Should you ever change it (or when you upgrade CFEngine), take special care to ensure the old and the new CFEngine can parse and execute this file successfully. If not, you risk losing control of your system (that is, if CFEngine cannot successfully execute update.cf, it has no mechanism for distributing new policy files).

By default, the policy defined in update.cf is executed at the beginning of a cf-execd scheduled agent run (see schedule and exec_command as defined in body executor control in controls/cf_execd.cf). When the update policy completes (regardless of success or failure) the policy defined in promises.cf is activated.

This is a standalone policy file. You can actually run it with cf-agent -KI -f ./update.cf but if you don't understand what that command does, please hold off until you've gone through the CFEngine documentation. The contents of update.cf duplicate other things under lib sometimes, in order to be completely standalone.

To repeat, when update.cf is broken, things go bonkers. CFEngine will try to run a backup failsafe.cf you can find in the C core under libpromises/failsafe.cf (that .cf file is written into the C code and can't be modified). If things get to that point, you probably have to look at why corrupted policies made it into production.

As is typical for CFEngine, the policy and the configuration are mixed. In controls/update_def.cf you'll find some very useful settings. Keep referring to controls/update_def.cf as you read this. We are skipping the nonessential ones.

Configure upstream masterfiles location for policy update

Want to get your policy from a place other than /var/cfengine/masterfiles on sys.policy_hub?

With an augments like this:

 code
{
  "vars": {
    "house": "Gryffindor",
    "mpf_update_policy_master_location": "/srv/cfengine/$(sys.flavor)/$(def.house)"
  }
}

A CentOS 7 host would copy policy from /srv/cfengine/centos_6/Gryffindor to $(sys.inputdir) (commonly /var/cfengine/inputs).

History:

Introduced in 3.12.0.

Add additional policy files for update (inputs)

You can append to the inputs used by the update policy via augments by defining vars.update_inputs. The following example will add the policy file my_updatebundle1.cf to the list of policy file inputs during the update policy.

 code
{
  "vars": {
    "update_inputs": [ "my_updatebundle1.cf" ]
  }
}

Evaluate additional bundles during update (bundlesequence)

You can specify bundles which should be run at the end of the default update policy bundlesequence by defining control_common_update_bundlesequence_end in the vars of an augments file.

For example:

 code
{
  "vars": {
    "control_common_update_bundlesequence_end": [ "my_updatebundle1", "mybundle2" ]
  }
}

Notes:

The order in which bundles are actuates is not guaranteed.
The agent will error if a named bundle is not part of inputs.

Specify the agent bundle used for policy update

The MPF uses cfe_internal_update_policy_cpv to update inputs and modules on remote agents. When new policy is verified by the agent /var/cfengine/masterfiles/cf_promises_validated is updated with the current timestamp. This file is used by remote agents to avoid unnecessary inspection of all files each time the update policy is triggered.

Override this bundle by setting def.mpf_update_policy_bundle via augments:

 code
{
  "variables": {
    "default:def.mpf_update_policy_bundle": {
      "value": "default:MyCustomPolicyUpdateBundle"
    }
  }
}

NOTE: Be sure to specify the namespace the bundle is in, for example, default.

History:

Introduced in 3.12.0

Ignore missing bundles

This option allows you to ignore errors when a bundle specified in body common control bundlesequence is not found.

This example illustrates enabling the option via augments.

 code
{
  "vars": {
    "control_common_ignore_missing_bundles": "true"
  }
}

NOTE: The same augments key is used for both update.cf and promsies.cf entries.

History:

Introduced in 3.12.0

Ignore missing inputs

This option allows you to ignore errors when a file specified in body common control inputs is not found.

This example illustrates enabling the option via augments.

 code
{
  "vars": {
    "control_common_ignore_missing_inputs": "true"
  }
}

NOTE: The same augments key is used for both update.cf and promsies.cf entries.

History:

Introduced in 3.12.0

Verify update transfers

Enable additional verrification after file transfers during policy update by defining the class cfengine_internal_verify_update_transfers. When this class is defined, the update policy will hash the transfered file and compare it against the hash given by the server

This augments file will enable this behavior for all clients.

 code
{
  "classes": {
    "cfengine_internal_verify_update_transfers": [ "any" ]
  }
}

Encrypted transfers

Note: When using protocol version 2 or greater all communications are encapsulated within a TLS session. This configuration option is only relevant for clients using protocol version 1 (default for versions 3.6 and prior).

To enable encryption during policy updates define the class cfengine_internal_encrypt_transfers.

Preserve permissions

By default the MPF does not enforce permissions of inputs. When masterfiles are copied to inputs, new files are created with default restrictive permissions. If the class cfengine_internal_preserve_permissions is defined the permissions of the policy server's masterfiles will be preserved when they are copied.

Enable CFEngine Enterprise HA

When the enable_cfengine_enterprise_hub_ha class is defined the policy to manage High Availability of Enterprise Hubs is enabled.

Note: This class is not defined by default.

Disable plain HTTP for CFEngine Enterprise Mission Portal

By default Mission Portal listens for HTTP requests on port 80, redirecting to HTTPS on port 443. To prevent the web server from listening on port 80 at all define default:cfe_cfengine_enterprise_disable_plain_http.

For example:

 code
{
  "classes": {
    "default:cfe_enterprise_disable_plain_http": {
      "class_expressions": [ "am_policy_hub|policy_server::" ]
    }
  }
}

Notes:

If this class (default:cfe_enterprise_disable_http_redirect_to_https) is defined the class default:cfe_enterprise_disable_plain_http is defined is automatically defined.

History:

Added in CFEngine 3.23.0, 3.21.3

Disable plain HTTP redirect to HTTPS for CFEngine Enterprise Mission Portal

By default Mission Portal listens for HTTP requests on port 80, redirecting to HTTPS on port 443. To prevent redirection of requests on HTTP to HTTPS define default:cfe_enterprise_disable_http_redirect_to_https.

For example:

 code
{
  "classes": {
    "default:cfe_enterprise_disable_http_redirect_to_https": {
      "class_expressions": [ "(am_policy_hub|policy_server).test_server::" ]
    }
  }
}

Notes:

If default:cfe_enterprise_disable_plain_http is defined, this class (default:cfe_enterprise_disable_http_redirect_to_https) is automatically defined.

History:

Added in CFEngine 3.6.0
Class renamed from cfe_cfengine_enterprise_enable_plain_http to cfe_enterprise_disable_http_redirect_to_https in CFEngine 3.23.0, 3.21.3

Disable cf_promises_validated check

For non policy hubs the default update policy only performs a full scan of masterfiles if cf_promises_validated is repaired. This repair indicates that the hub has validated new policy that the client needs to refresh.

To disable this check define the cfengine_internal_disable_cf_promises_validated class.

It not recommended to disable this check as it both removes a safety mechanism that checks for policy to be valid before allowing clients to download updates, and the increased load on the hub will affect scalability.

If you want to periodically perform a full scan consider adding custom policy to simply remove $(sys.inputdir)/cf_promises_validated. This will cause the file to be repaired during the next update run triggering a full scan.

Disable automatically removing files not present upstream (SYNC masterfiles)

By default, the MPF will keep inputdir in sync with masterfiles on the hub. If the class cfengine_internal_purge_policies_disabled is defined the update behavior will only keep files that exist on the remote up to date locally, files that exist locally that do not exist upstream will be left behind. Note, if this is disabled and a policy file that is dynamically loaded based on it's presence is renamed, duplicate definition errors may occur, preventing policy execution.

This augments file will enable this behavior for all clients.

 code
{
  "classes": {
    "cfengine_internal_purge_policies_disabled": [ "any" ]
  }
}

History:

Introduced in 3.18.0, previously, the default behavior was opposite and the class cfengine_internal_purge_policies had to be enabled to keep inputs in sync with masterfiles.

Disable limiting robot agents

By default the MPF (Masterfiles Policy Framework) contains active policy that is intended to remediate a pathological condition where multiple agent component daemons (like cf-execd) are running concurrently.

Define the class mpf_disable_cfe_internal_limit_robot_agents to disable this automatic remediation.

 code
{
  "classes": {
    "mpf_disable_cfe_internal_limit_robot_agents": [ "any" ]
  }
}

History:

Introduced in 3.15.0, 3.12.3, 3.10.7

Automatically deploy policy from version control

On a CFEngine Enterprise Hub during the update policy if the class cfengine_internal_masterfiles_update is defined masterfiles will be automatically deployed from an upstream version control repository using the settings defined via Mission Portal or directly in /opt/cfengine/dc-scripts.

Note: Any policy in the distribution location (/var/cfengine/masterfiles) will be deleted the first time this tooling runs. Be wary of local modifications before enabling.

Exclude files in policy analyzer

When the policy analyzer is enabled, a copy of the policy is made available for viewing from Mission Portal. To exclude files from this view you can define def.cfengine_enterprise_policy_analyzer_exclude_files as a list of regular expressions matching files that you do not want to be viewable from Policy Analyzer.

This augments file will prevent any files named please-no-copy and any file names that contain no-copy-me from being copied and visible from Policy Analyzer.

 code
{
  "vars": {
    "cfengine_enterprise_policy_analyzer_exclude_files": [ "please-no-copy", ".*no-copy-me.*" ]
  }
}

History:

Added in 3.19.0, 3.18.1

Policy permissions

By default the policy enforces permissions of 0600 meaning that inputs are only readable by their owner. If you are distributing scripts with your masterfiles, be sure there is a policy to ensure they are executable when you expect them to be.

Agent binary upgrades

Remote agents can upgrade their own binaries using the built in binary upgrade policy. Packages must be placed in /var/cfengine/master_software_updates in the appropriate platform directory. Clients will automatically download and install packages when the trigger_upgrade class is defined during a run of update.cf.

This augments file will defines trigger_upgrade on hosts with IPv4 addresses in 192.0.2.0/24 or 203.0.113.0/24 or hosts that have a cfengine 3.10 class not for cfengine 3.10.2.

 code
{
   "classes": {
    "trigger_upgrade": [
      "ipv4_10_10_1",
      "ipv4_10_10_2",
      "cfengine_3_10_(?!2$)\\d+"
    ]
   }
}

Notes:

This policy is specific to CFEngine Enterprise.
The negative look ahead regular expression is useful because it automatically turns off on hosts after they reach the target version.

Configure path that software is served from for autonomous agent upgrades

def.master_software_updates defines the path that cfengine policy servers share software updates from. Remote agents access this path via the master_software_updates shortcut. By default this path is $(sys.workdir)/master_software_updates. This path can be overridden via vars.dir_master_software_updates in augments.

For example:

 code
{
  "vars": {
    "dir_master_software_updates": "/srv/cfengine-software-updates/"
  }
}

History:

Introduced 3.15.0, 3.12.3, 3.10.8

Disable seeding binaries on hub

By default when trigger_upgrade is defined on a hub, the hub will download packages for agents to use during self upgrade. This automatic download behavior is disabled when the class mpf_disable_hub_masterfiles_software_update_seed is defined.

For example:

 code
{
  "classes": {
    "mpf_disable_hub_masterfiles_software_update_seed": [ "policy_server::" ]
  }
}

History:

Introduced 3.19.0, 3.18.1

Override files considered for copy during policy updates

The default update policy only copies files that match regular expressions listed in def.input_name_patterns.

This augments file ensures that only files ending in .cf, .dat, .mustache, .json, .yaml and the file cf_promises_release_id will be considered by the default update policy.

 code
{
  "vars": {
    "input_name_patterns": [
      ".*\\.cf", ".*\\.dat", ".*\\.txt", ".*\\.conf",
      ".*\\.mustache", ".*\\.sh", ".*\\.pl", ".*\\.py", ".*\\.rb",
      ".*\\.sed", ".*\\.awk", "cf_promises_release_id", ".*\\.json",
      ".*\\.yaml", ".*\\.csv"
    ]
  }
}

Note: This filter does not apply to bootstrap operations. During bootstrap the embedded failsafe policy is used and it decides which files should be copied.

Extend files considered for copy during policy updates

The default update policy only copies files that match regular expressions listed in default:def.input_name_patterns. The variable default:update_def.input_name_patterns allows the definition of additional filename patterns without having to maintain the full set of defaults.

This augments file additionally ensures that files ending in .tpl, .md, and .org are also copied.

 code
{
  "variables": {
    "default:update_def.input_name_patterns_extra": {
      "value": [ ".*\\.tpl", ".*\\.md", ".*\\.org" ],
      "comment": "We use classic CFEngine templates suffixed with .tpl so they should be copied along with documentation."
    }
  }
}

Note: This filter does not apply to bootstrap operations. During bootstrap the embedded failsafe policy is used and it decides which files should be copied.

History:

Introduced in CFEngine 3.23.0, 3.21.3

Configuring component management

The Masterfiles Policy Framework ships with policy to manage the components of CFEngine.

By default, for hosts without systemd, this policy defaults to ensuring that components are running.

On systemd hosts, the policy to manage component units is disabled by default.

Enable management of components on systemd hosts

To allow the Masterfiles Policy Framework to actively manage cfengine systemd units and state define the mpf_enable_cfengine_systemd_component_management.

This example illustrates enabling management of components on systemd hosts having a class matching redhat_8 via augments.

 code
{
  "classes": {
    "mpf_enable_cfengine_systemd_component_management": [ "redhat_8" ]
  }
}

When enabled, the policy will render systemd unit files in /etc/systemd/system for managed services. Mustache templates for service units are in the templates directory in the root of the Masterfiles Policy Framework.

When enabled, the policy will make sure that all units are enabled, unless they have been disabled by a persistent class or are explicitly listed as an agent to be disabled.

Class: default:persistent_disable_DAEMON

Description: Disable a CFEngine Enterprise daemon component persistently.

DAEMON can be one of cf_execd, cf_monitord or cf_serverd.

This will stop the AGENT from starting automatically.

This augments file will ensure that cf-monitord is disabled on hosts that have server1 or the redhat class defined.

 code
{
  "classes": {
    "persistent_disable_cf_monitord": [ "server1", "redhat" ]
  }
}

Class: clear_persistent_disable_DAEMON

Description: Re-enable a previously disabled CFEngine Enterprise daemon component.

DAEMON can be one of cf_execd, cf_monitord or cf_serverd.

This augments file will ensure that cf-monitord is not disabled on redhat hosts.

 code
{
  "classes": {
    "clear_persistent_disable_cf_monitord": [ "redhat" ]
  }
}

Variable: default:def.agents_to_be_disabled

Description: list of agents to disable.

This augments file is a way to specify that cf-monitord should be disabled on all hosts.

 code
{
  "vars": {
    "agents_to_be_disabled": [ "cf-monitord" ]
  }
}

Main policy (promises.cf)

The following settings are defined in controls/def.cf can be set from an augments file.

Automatically migrate ignore_interfaces.rx to workdir

ignore_interfaces.rx defines regular expressions matching network interfaces that CFEngine should ignore.

Prior to 3.23.0 this file was expected to be found in $(sys.inputdir)/ignore_interfaces.rx. Beginning with 3.23.0 preference is given to $(sys.workdir)/ignore_interfaces.rx if it is found. A WARNING is emitted by cfengine if the file is found only in $(sys.inputdir).

When the class default:mpf_auto_migrate_ignore_interfaces_rx_to_workdir is defined (not defined by default) $(sys.workdir)/ignore_interfaces.rx is maintained as a copy of $(sys.inputdir)/ignore_interfaces.rx.

 code
{
  "classes": {
    "default:mpf_auto_migrate_ignore_interfaces_rx_to_workdir": {
      "class_expressions": [ "cfengine_3_23|cfengine_3_24::" ],
      "comment": "Automatically migrate ignore_interfaces.rx to workdir."
    }
  }
}

Additionally, to disable reports about the presence of $(sys.inputdir)/ignore_interfaces.rx define the class default:mpf_auto_migrate_ignore_interfaces_rx_to_workdir_reports_disabled. When this class is not defined, cf-agent will emit reports indicating it's presence and state in relation to $(sys.workdir)/ignore_interfaces.rx.

 code
{
  "classes": {
    "default:mpf_auto_migrate_ignore_interfaces_rx_to_workdir_reports_disabled": {
      "class_expressions": [ "cfengine_3_23|cfengine_3_24::" ],
      "comment": "We don't want reports about legacy ignore_interfaces.rx to be emitted."
    }
  }
}

History:

Introduced default:mpf_auto_migrate_ignore_interfaces_rx_to_workdir and default:mpf_auto_migrate_ignore_interfaces_rx_to_workdir_reports_disabled in 3.23.0, 3.21.4

dmidecode inventory

When dmidecode is present, some key system attributes are inventoried. The inventoried attributes can be overridden by defining def.cfe_autorun_inventory_demidecode[dmidefs] via augments. dmidecode queries each key in dmidefs and tags the result with the value prefixed with attribute_name= Note, as the dmidefs are overridden, you must supply all desired inventory attributes.

For example:

 code
{
  "vars": {
    "cfe_autorun_inventory_dmidecode": {
      "dmidefs": {
        "bios-vendor": "BIOS vendor",
        "bios-version": "BIOS version",
        "system-serial-number": "System serial number",
        "system-manufacturer": "System manufacturer",
        "system-version": "System version",
        "system-product-name": "System product name",
        "bios-release-date": "BIOS release date",
        "chassis-serial-number": "Chassis serial number",
        "chassis-asset-tag": "Chassis asset tag",
        "baseboard-asset-tag": "Baseboard asset tag"
      }
    }
  }
}

History:

Introduced 3.13.0, 3.12.1, 3.10.5

Configure proc inventory

By default the MPF inventories consoles, cpuinfo, modules, partitions, and version from /proc. This can be adjusted by defining default:cfe_autorun_inventory_proc.basefiles.

For example:

 code
{
  "variables": {
    "default:cfe_autorun_inventory_proc.basefiles": {
      "value": [
        "consoles",
        "cpuinfo",
        "version"
      ],
      "comment": "We do not need the extra variables this produces since we get the info differently",
      "tags": [
        "Custom override MPF default"
      ]
    }
  }
}

History:

Added 3.21.0

Configure cf-agent syslog facility

To configure the syslog facility used by cf-agent configure agentfacility by setting default:def.control_agent_agentfacility via augments to one of the allowed values (LOG_USER, LOG_DAEMON, LOG_LOCAL0, LOG_LOCAL1, LOG_LOCAL2, LOG_LOCAL3, LOG_LOCAL4, LOG_LOCAL5, LOG_LOCAL6, LOG_LOCAL7)

 code
{
  "variables": {
    "default:def.control_agent_agentfacility": {
      "value": "LOG_USER"
    }
  }
}

History:

Added in 3.22.0, 3.21.2