The Complete Guide
Table of Content
- Introduction and System Overview
- Latest Release
- Installation and Configuration
- Writing and Serving Policy
- Language Concepts
- Design Center Overview
- Reporting
- FAQ
- Additional Resources
CFEngine is a configuration management system that provides a framework for automated management of IT infrastructure.
CFEngine is decentralized and highly scalable. It is powered by autonomous agents that can continuously monitor, self-repair, and update or restore an entire IT system, with negligible impact on system resources or performance.
See Also: Introduction and System Overview
CFEngine Features
- Defines the configuration of an entire IT system, including: Devices, Users, Applications, and Services.
- Helps maintain that system over time.
- Checks the system state at any given moment.
- Ensures compliance with a desired system state.
- Propagates real-time modifications or updates across the system.
Choose a CFEngine Version
CFEngine Enterprise is a licensed edition for enterprises that plan to use the tool in production environments. The Enterprise edition comes in several variants, including one that can be evaluated for free (up to 25 servers).
CFEngine Community, a free GPL v3 open source edition.
See also:
Install It
There are several steps to bring up a CFEngine installation within an organization:
- Prepare all appropriate machines for installation.
- Configure your network and security.
- Download the CFEngine software.
- Install CFEngine on the Policy Server(s).
- Bootstrap the Policy Server to itself.
- Initiate post-install configuration on the Policy Server.
- Install CFEngine on the Host machine(s).
- Bootstrap the Host(s) to a Policy Server.
See Installation and Configuration for a more detailed guide on how to get CFEngine up and running for various environments.
Try It
Walk through the examples, tutorials and how to guides to get a better feel for the power and value of CFEngine:
Learn More
Take a look at the CFEngine Guide to learn more about CFEngine's architecture and components, as well as how to write policy that can help manage IT systems.
Check out Additional Resources and the CFEngine Learning Center, for more guides, demos, and other resources from our CFEngine staff and our special CFEngine contributors.
Use our Help
Support and Community We provide a number of ways to connect you to CFEngine experts if you need more help. Contact us!
CFEngine Guide
- Introduction and System Overview
- Latest Release
- Installation and Configuration
- Pre-Installation Checklist
- General Installation
- [Upgrading to 3.7]
- Secure Bootstrap
- Writing and Serving Policy
- Reports
- Design Center Overview
- FAQ
- Additional Resources
Introduction and System Overview
CFEngine is a distributed system for managing and monitoring computers across an IT network. Machines on the network that have CFEngine installed, and have registered themselves with a policy server (see Installation and Configuration), will each be running a set of CFEngine component applications that manage and interpret textual files called policies. Policy files themselves contain sets of instructions to ensure machines on the network are in full compliance with a defined state. At the atomic level are sets, or bundles, of what are known in the CFEngine world as Promises. Promises are at the heart of Promise Theory, which is in turn what CFEngine is all about.
- Policy Language and Compliance
- CFEngine Policy Servers and Hosts
- CFEngine Component Applications and Daemons
Policy Language and Compliance
For many users, CFEngine is simply a configuration tool – i.e. software for deploying and patching systems according to a policy. Policy is described using promises. Every statement in CFEngine 3 is a promise to be kept at some time or location. More than this, however, CFEngine is not like other automation tools that "roll out" an image of some software once and hope for the best. Every promise that you make in CFEngine is continuously verified and maintained. It is not a one-off operation, but a self-repairing process should anything deviate from the policy.
CFEngine ensures that the actual state of a system is in compliance with the predefined model of desired state for the system. If it is not in compliance CFEngine will bring it into compliance. This is known as convergence.
That model is represented by one or more policies that have been written using the declarative CFEngine policy language. The policy language has been designed with a vocabulary that is intuitive, yet at the same time can still support the design of highly complex IT systems.
Those policies are distributed across all hosts within the system via download from the policy server. Every host will then interpret and execute each of the instructions it has been given in a predetermined order.
CFEngine continually monitors all of the hosts in real-time, and should the system’s current state begin to drift away from the intended state then CFEngine will automatically take corrective action to bring everything back into compliance.
See Also: Language Concepts, Writing and Serving Policy
CFEngine Policy Servers and Hosts
There are basically two categories of machines in a CFEngine environment: policy servers and their client hosts. Policy servers are responsible for making policy files available to each of the client hosts that have registered with it (a.k.a. bootstrapped), including itself. Hosts on the other hand are responsible for ensuring they continuously pull in the latest policies, or changes to policies, from the policy server. They are additionally responsible for ensuring they remain fully compliant with the instructions contained within the policy files, at all times.
The role of a particular machine where CFEngine is deployed determines which of the components will be installed and running at any given moment.
See Also: Writing and Serving Policy
CFEngine Component Applications and Daemons
There are a number of components in CFEngine, with each component performing a unique function: components responsible for implementing promises, components responsible for organizing large networks of agents, and other components responsible for providing the infrastructure of CFEngine.
These components form the basis of automation with CFEngine. They are independent software agents running on the various systems that make up your infrastructure. They communicate with one another as shown in the following figure, using a protocol that allows each host to distribute promises, act upon them, and report status to a central server.
All CFEngine software components exist in /var/cfengine/bin
.
Daemons
All machines, whether they are policy servers or hosts, will have these three important daemons running at all times:
cf-execd
cf-execd
is the scheduling daemon for cf-agent
, similar to cron. It executes and collects the output of cf-agent
and
e-mails any output to the configured e-mail address.
cf-execd
runs cf-agent
locally according to a schedule specified in policy code (executor control body). After a cf-agent
run is completed, cf-execd
gathers output from cf-agent
, and may be configured to email the output to a specified address. It may also be configured to splay
(randomize) the execution schedule to prevent synchronized cf-agent
runs across a network.
cf-execd
keeps the promises made in common bundles, and is affected by common and executor control bodies.
See also: cf-execd reference documentation.
cf-serverd
cf-serverd
is a socket listening daemon providing two services: it acts as a file server for remote file copying and it allows an authorized cf-runagent
to start a cf-agent
run. cf-agent
typically connects to a cf-serverd
instance to request updated policy code, but may also request additional files for download. cf-serverd
employs role based access control (defined in policy code) to authorize requests.
cf-serverd
keeps the promises made in common
and server
bundles, and is affected by common
and server
control bodies.
By starting this daemon you can set up a line of communication between hosts. The server is able to share files and receive requests to execute existing policy on an individual machine. It is not possible to send (push) new information to CFEngine from outside.
This daemon authenticates requests from the network and processes them according to rules specified in the server control body and server bundles containing access promises.
See also: cf-serverd reference documentation.
cf-monitord
cf-monitord
is the monitoring daemon for CFEngine. It samples probes defined in policy using measurements type promises and attempts to learn the normal system state based on current and past observations. Current estimates are made available as special variables (e.g. $(mon.av_cpu)) to cf-agent
, which may use them to inform policy decisions.
cf-monitord
keeps the promises made in common
and monitor
bundles, and is affected by common
and monitor
control bodies.
See also: cf-monitord reference documentation.
Other Component Applications
- /var/cfengine/bin/cf-agent
- /var/cfengine/bin/cf-key
- /var/cfengine/bin/cf-promises
- /var/cfengine/bin/cf-runagent
cf-agent
cf-agent
evaluates policy code and makes changes to the system. Policy bundles are evaluated in the order of the provided bundlesequence
(this is normally specified in the common control body). For each bundle, cf-agent groups promise statements according to their type. Promise types are then evaluated in a preset order to ensure fast system convergence to policy.
cf-agent
keeps the promises made in common
and agent
bundles, and is affected by common
and agent
control bodies.
cf-agent
is the instigator of change. Everything that happens on a client machine
happens because of cf-agent
. The agent is the part of CFEngine that
manipulates system resources.
cf-agent
's only contact with the network is via remote copy requests. It
does not and cannot grant any access to a system from the network. It is only
able to request access to files from the server component.
See also: cf-agent
reference documentation.
cf-key
The CFEngine key generator makes key pairs for remote authentication.
See also: cf-key
reference documentation.
cf-promises
cf-promises
is CFEngine's promise verifier. It is used to run a "pre-check" of
policy code before cf-agent
attempts to execute.
cf-promises
operates by first parsing policy code checking for syntax errors. Second, it validates the integrity of policy consisting of multiple files. Third, it checks for semantic errors, e.g. specific attribute set rules. Finally, cf-promises
attempts to expose errors by partially evaluating the policy, resolving as many variable and classes promise statements as possible. At no point does cf-promises
make any changes to the system.
In 3.6.0 and later, cf-promises
will not evaluate function calls either. This may affect customers who use execresult for instance. Use the new --eval-functions yes command-line option (default is no) to retain the old behavior from 3.5.x and earlier.
See also: cf-promises
reference documentation.
cf-runagent
cf-runagent
is a helper program that can be used to run cf-agent
on a number of remote
hosts. It cannot be used to tell cf-agent
what to do, it can only ask
cf-serverd
on the remote host to run the cf-agent
with its existing
policy. It can thus be used to trigger an immediate deployment of new policy,
if their existing policy includes that they check for updates.
Privileges can be granted to users to provide a kind of Role Based Access Control (RBAC) to certain parts of the existing policy.
cf-runagent
connects to a list of running instances of cf-serverd
. It allows foregoing the usual cf-execd
schedule to activate cf-agent
. Additionally, a user may send classes to be defined on the remote host. Two kinds of classes may be sent: classes to decide on which hosts cf-agent
will be started, and classes that the user requests cf-agent
should define on execution. The latter type is regulated by cf-serverd
's role based access control.
See also: cf-runagent
reference documentation.
CFEngine Architecture and Design
CFEngine operates autonomously in a network, under your guidance. While CFEngine supports anything from 1 servers to 100,000+ servers, the essence of any CFEngine deployment is the same.
CFEngine supports networks of any size, from a handful of nodes to hundreds of thousands of computers. It is built to scale. If your site is very large (many thousands of servers) you should spend some time discussing your requirements with CFEngine experts. They will know how to tune promises and configurations to your environment as scale requires you to have more infrastructure, and a potentially more complicated configuration. No matter the scale, the essence of any CFEngine deployment is the same, but with great power comes great responsibility (a.k.a. don't break things before the weekend, on the weekend, or in fact on any other day).
CFEngine was designed to enable scalable configuration management in any kind of environment, with an emphasis on supporting large, Unix-like systems that are connected via TCP/IP.
CFEngine doesn't depend on or assume the presence of reliable infrastructure. It works opportunistically in any environment, using the fewest possible resources, and it has a limited set of software dependencies. It can run anywhere and this lean approach to CFEngine's architecture makes it possible to support both traditional server-based approaches to configuration as well as more novel platforms for configuration including embedded and mobile systems.
CFEngine's design allows you to create fault-tolerant, available systems which are independent of external requirements. CFEngine works in all the places you think it should, and all the new places you haven't even thought of yet.
Managing Expectations with Promises
CFEngine works on a simple notion of promises. A promise is the documentation of an intention to act or behave in some manner. When you make a promise, it is an effort to improve trust. Trust is an economic time-saver. If you can't trust you have to verify everything, and that is expensive.
Everything in CFEngine can be thought of as a promise to be kept by different
resources in the system. In a system that delivers a web site with Apache
httpd
, an important promise may be to make sure that the httpd
or apache
package is installed,
running, and accessible on port 80. In a system which needs to satisfy mid-day
traffic on a busy web site, a promise may be to ensure that there are 200
application servers running during normal business hours.
These promises are not top-down directives for a central authority to push through the system. A large organization can't run on top-down authority alone. A group of people can't be managed without empowering and trusting them to make independent decisions.
CFEngine is a system that emphasizes the promises a client makes to the overall CFEngine network. They are the rules which clients are responsible for implementing. We can create large systems of scale because we don't create a bulky centralized authority. There should be no single point-of-failure when managing machines and people.
Combining promises with patterns to describe where and when promises should apply is what CFEngine is all about.
Automation with CFEngine
Users are good at researching solutions and making design decisions, but awful at repeated execution. Machines are pitiful at making decisions, but very good at reliable implementation at very large scale. It makes sense to let each side do the job that they are good at. With CFEngine, users make decisions and write promises for machines to implement and satisfy.
A CFEngine user will declare a promise in CFEngine, and CFEngine will then translate this promise into a series of actions to implement. For the most part, CFEngine understands how to deliver on promises, and they don't need to be given explicit instructions for completing tasks. It is your job to make decisions about the systems you are managing and to describe those in suitable promises. It is CFEngine's job to automate and deliver a promise.
CFEngine is a distributed solution that is completely independent of host operating systems, network topology or system processes. You describe the ideal state of a given system by creating promises and the CFEngine agents ensures that the necessary steps are taken to achieve this state. Automation in CFEngine is executed through a series of components that run locally on hosts.
Phases of System Management
There are four commonly cited phases in managing systems with CFEngine: Build, Deploy, Manage, and Audit.
Build
A system is based on a number of decisions and resources that need to be `built' before they can be implemented. You don't need to decide every detail, just enough to build trust and predictability into your system. In CFEngine, what you build is a template of proposed promises for the machines being managed. If the machines in a system all make and keep these promises, the system will function seamlessly as planned.
Deploy
Deploying really means implementing the policy that was already decided. In transaction systems, one tries to push out changes one-by-one, hence `deploying' the decision. In CFEngine you simply publish your policy (in CFEngine parlance these are "promise proposals") and the machines see the new proposals and can adjust accordingly. Each machine runs an agent that is capable of implementing policies and maintaining them over time without further assistance.
Manage
Once a decision is made, unplanned events will occur. Such incidents traditionally set off alarms and humans rush to make new transactions to repair them. In CFEngine, the autonomous agent manages the system, and you only have to deal with rare events that cannot be dealt with automatically. This is the key difference of CFEngine, a focus on autonomy and creating agents that are smart enough to adapt to changing situations.
Audit
In traditional configuration systems, the outcome is far from clear after a one-shot transaction, so one audits the system to determine what actually happened. In CFEngine, changes are not just initiated once, but locally audited and maintained. Decision outcomes are assured by design in CFEngine and maintained automatically, so the main worry is managing conflicting. Users can sit back and examine regular reports of compliance generated by the agents, without having to arrange for new transactions to roll-out changes.
You should not think of CFEngine as a roll-out system, i.e. one that attempts to force out absolute changes and perhaps reverse them in case of error. Roll-out and roll-back are theoretically flawed concepts that only sometimes work in practice. With CFEngine, you publish a sequence of policy revisions, always moving forward (because like it or not, time only goes in one direction). All of the desired-state changes are managed locally by each individual host, and continuously repaired to ensure on-going compliance with policy.
See Also: Networking
CFEngine Directory Structure
The CFEngine application is fully contained within the /var/cfengine directory tree. Here is a quick breakdown of the directory structure and some of the files and functions associated with each subdirectory.
- /var/cfengine/bin
- Directories for Policy Files
- Output Directories
- Log Files in /var/cfengine
- Database Files in /var/cfengine
- Process (AKA PID) Files in /var/cfengine
- Sockets in /var/cfengine
- Datafiles in /var/cfengine
- Binary Files in /var/cfengine
- git in /var/cfengine/bin
- Misc. in /var/cfengine/bin
- Postgres in /var/cfengine/bin
- Redis in /var/cfengine/bin
/var/cfengine/bin
Agents
cf-agent
: Executes the promises.cf file; ensures that all promises are being keptcf-consumer
cf-hub
cf-key
cf-promises
: Verifies CFEngine's configuration syntaxcf-runagent
: Contacts a remote system to run cf-agentcf-twin
Daemons
cf-execd
: Starts the cf-agent process at a specified time interval.cf-monitord
: Collects system statisticscf-serverd
: Provides network services; used to distribute policy and data files
See Also: CFEngine Component Applications and Daemons
Directories for Policy Files
/var/cfengine/modules
Location of scripts used in commands
promises.
/var/cfengine/inputs
Cached policy repository on each CFEngine client. When cf-agent
is
invoked by cf-execd
, it reads only from this directory.
/var/cfengine/masterfiles
Policy repository which grants access to local or bootstrapped CFEngine
clients when they need to update their policies. Policies obtained from
/var/cfengine/masterfiles
are then cached in /var/cfengine/inputs
for
local policy execution. The cf-agent
executable does not execute policies
directly from this repository.
Output Directories
/var/cfengine/outputs
Directory where cf-agent
creates its output files. The outputs directory is
a record of spooled run-reports. These are often mailed to the administrator
by cf-execd
, or can be copied to another central location and viewed in an
alternative browser. However, not all hosts have an email capability or are
online, so the reports are kept here.
/var/cfengine/reports
Directory used to store reports. Reports are not tidied automatically, so you should delete these files after a time to avoid a build up.
/var/cfengine/state
State data such as current process identifiers of running processes, persistent classes and other cached data.
/var/cfengine/lastseen
Log data for incoming and outgoing connections.
/var/cfengine/cfapache
/var/cfengine/config
/var/cfengine/design-center
/var/cfengine/httpd
/var/cfengine/lib
Directory to store shared objects and dependencies that are in the bundled packages.
/var/cfengine/lib-twin
/var/cfengine/master_software_updates
/var/cfengine/plugins
/var/cfengine/ppkeys
Directory used to store encrypted public/private keys for CFEngine client/server network communications.
/var/cfengine/share
/var/cfengine/software_updates
/var/cfengine/ssl
Log Files in /var/cfengine
On hosts, CFEngine writes numerous logs and records to its private workspace.
CFEngine Enterprise provides solutions for centralization and network-wide reporting at an arbitrary scale.
cf3.[hostname].runlog
A time-stamped log of when each lock was released. This shows the last time each individual promise was verified.
cfagent.[hostname].log
Although ambiguously named (for historical reasons) this log contains the current list of setuid/setgid programs observed on the system. CFEngine warns about new additions to this list. This log has been deprecated.
cf_notkept.log
In CFEngine Enterprise, a list of promises, with handles and comments, that were not kept.
cf_repair.log
In CFEngine Enterprise, a list of promises, with handles and comments, that were repaired.
promise_summary.log
A time-stamped log of the percentage fraction of promises kept after each run.
Database Files in /var/cfengine
- bundles.lmdb
cf_classes.lmdb
A database of classes that have been defined on the current host, including their relative frequencies, scaled like a probability.
cf_lastseen.lmdb
A database of hosts that last contacted this host, or were contacted by this host, and includes the times at which they were last observed.
checksum_digests.lmdb
The database of hash values used in CFEngine's change management functions.
nova_agent_execution.lmdb
nova_track.lmdb
performance.lmdb
A database of last, average and deviation times of jobs recorded by
cf-agent
. Most promises take an immeasurably short time to check, but
longer tasks such as command execution and file copying are measured by
default. Other checks can be instrumented by setting a
measurement_class
in the action
body of a promise.
Process (AKA PID) Files in /var/cfengine
The CFEngine components keep their current process identifier number in `pid files' in the work directory.
cf-consumer.pid
cf-execd.pid
cf-hub.pid
cf-monitord.pid
cf-serverd.pid
Sockets in /var/cfengine
cf-hub-local
Datafiles in /var/cfengine
policy_server.dat
IP address of the policy server
Binary Files in /var/cfengine
randseed
git in /var/cfengine/bin
bin/git
bin/git-cvsserver
bin/gitk
bin/git-receive-pack
bin/git-shell
bin/git-upload-archive
bin/git-upload-pack
Misc. in /var/cfengine/bin
bin/curl
bin/lmdump
bin/openssl
bin/rpmvercmp
bin/rsync
bin/runalerts.sh
Postgres in /var/cfengine/bin
bin/clusterdb
bin/createdb
bin/createlang
bin/createuser
bin/dropdb
bin/droplang
bin/dropuser
bin/initdb
bin/pg_basebackup
bin/pg_config
bin/pg_controldata
bin/pg_ctl
bin/pg_dump
bin/pg_dumpall
bin/pg_isready
bin/pg_receivexlog
bin/pg_resetxlog
bin/pg_restore
bin/postgres
bin/postmaster
bin/psql
bin/reindexdb
bin/vacuumdb
Redis in /var/cfengine/bin
bin/redis-benchmark
bin/redis-check-aof
bin/redis-check-dump
bin/redis-cli
bin/redis-server
Not Verified
state/cf_lock.lmdb
A database of active and inactive locks and their expiry times. Deleting this database will reset all lock protections in CFEngine.
state/history.lmdb
CFEngine Enterprise maintains this long-term trend database.
state/cf_observations.lmdb
This database contains the current state of the observational history of
the host as recorded by cf-monitord
.
state/cf_state.lmdb
A database of persistent classes active on this current host.
state/nova_measures.lmdb
CFEngine Enterprise database of custom measurements.
state/nova_static.lmdb
CFEngine Enterprise database of static system discovery data.
state/cf_procs
A cache of the process table. This is useful formeasurement
promises about processes.state/cf_rootprocs
A cache of the process table of processes owned by the root user. This is useful formeasurement
promises about processes.state/cf_otherprocs
A cache of the process table for processes not owned by the root user. This is useful formeasurement
promises about processes.state/file_changes.log
A time-stamped log of which files have experienced content changes since the last observation, as determined by the hashing algorithms in CFEngine.
state/*_measure.log
CFEngine Enterprise maintains user-defined logs based on specifically promised observations of the system.
state/env_data
This file contains a list of currently discovered classes and variable values that characterize the anomaly alert environment. They are altered by the monitor daemon.
/var/logs/CFEngineHub-Install.log
This file contains logs related to the CFEngine package installation.
Networking
Starting cf-serverd
sets up a line of communication between
hosts. This daemon authenticates requests from the network and processes them
according to rules specified in the
server control
body and server bundles
containing access
promises.
The server can allow the network to access files or to execute CFEngine:
The only contact
cf-agent
makes to the server is via remote copy requests. It does not and cannot grant any access to a system from the network. It is only able to request access to files on the remote server.cf-runagent
can be used to runcf-agent
on a number of remote hosts.
Unlike other approaches to automation, CFEngine does not rely on SSH key authentication and configuring trust, the communication between hosts is very structured and also able to react to availability issues. This means that you don't have to arrange login credentials to get CFEngine to work. If large portions of your network stop working, then CFEngine on each individual host understands how to keep on running and delivering promises.
In particular, if the network is not working, CFEngine agents skip downloading new promises and continue with what they already have. CFEngine was specifically designed to be resilient against connectivity issues network failure may be in question. CFEngine is fault tolerant and opportunistic.
Connecting to server
In order to connect to the CFEngine server you need:
- A public-private key pair. It is automatically generated during package
installation or during bootstrap. To manually create a key pair,
run
cf-key
. - Network connectivity with an IPv4 or IPv6 address.
- Permission to connect to the server.
The
server control
body must grant access to your computer and public key by name or IP address, by listing it in the appropriate access lists (see below). - Mutual key trust.
Your public key must be trusted by the server, and you must trust the server's
public key. The first part is established by having the
trustkeysfrom
setting open on the server for the first connection of the agent. It should be closed later to avoid trusting new agents. The second part is established by bootstrapping the agent to the hub, or by executing acopy_from
files promise usingtrustkey=>"true"
. - Permission to access something.
Your host name or IP address must be mentioned in an
access
promise inside a server bundle, made by the file that you are trying to access.
If all of the above criteria are met, connection will be established and data will be transferred between client and server. The client can only send short requests, following the CFEngine protocol. The server can return data in a variety of forms, usually files, but sometimes console output.
Bootstrapping
Bootstrap is the manual first run of cf-agent that establishes
communication with the policy server.
Bootstrapping executes the failsafe.cf
policy that connects to
the server, establishes trust to the server's key, and that starts the
CFEngine daemon processes cf-execd
, cf-serverd
and cf-monitord
.
The host that other hosts are bootstrapped to
automatically assumes the role of policy server.
You should bootstrap the policy server first to itself:
$ /var/cfengine/bin/cf-agent --bootstrap [public IP of localhost]
Then execute the same step (using the exact same IP) on all hosts that should pull policy from that server. CFEngine will create keys if there are none present, and exchange those to establish trust.
CFEngine will output diagnostic information upon bootstrap. In case of error,
investigate the access
promises the server is making (run cf-serverd
in
verbose mode on the policy hub for more informative messages). Note that
by default, CFEngine's server daemon cf-serverd
trusts incoming connections
from hosts within the same /16
subnet.
After a host has been bootstrapped, the text file policy_server.dat
in
the CFEngine installation contains the IP address of the policy server.
Key exchange
The key exchange model used by CFEngine is based on that used by OpenSSH. It is a peer to peer exchange model, not a central certificate authority model. This means that there are no scalability bottlenecks (at least by design, though you might introduce your own if you go for an overly centralized architecture).
Key exchange is handled automatically by CFEngine and all you need to do is to
decide which keys to trust. The server (cf-serverd
) trusts new keys only
from addresses in trustkeysfrom
. Once a key has been
accepted you should close down trustkeysfrom
list. Then, even if a malicious peer
is spoofing an allowed IP address, its unknown key will be denied.
Once you have arranged for the right to connect to the server, you must decide
which hosts will have access to which files. This is done with access
promises.
bundle server access_rules()
{
access:
"/path/file"
admit => { "127.0.0.1", "127.0.0.2", "127.0.0.3" },
deny => { "192.168.0.0/8" };
}
On the client side, i.e. cf-runagent
and cf-agent
, there are three issues:
- Choosing which server to connect to.
- Trusting the key of any previously unknown servers
- Choosing whether data transfers should be encrypted (with
encrypt
) - not applicable if you are using newprotocol_version
.
There are two ways of managing trust of server keys by a client. One is an
automated option, setting the option trustkey
in a copy_from
files promise, e.g.
body copy_from example
{
# .. other settings ..
trustkey => "true";
}
Another way is to run cf-runagent
in interactive mode. When you run
cf-runagent
, unknown server keys are offered to you interactively (as with
ssh
) for you to accept or deny manually:
$ WARNING - You do not have a public key from host ubik.iu.hio.no = 128.39.74.25
$ Do you want to accept one on trust? (yes/no)
-->
Once public keys have been exchanged from client to server and from server to client, the issue of trust is solved according to public key authentication schemes. You only need to worry about trust when one side of a connection has never seen the other side before.
Time windows (races)
All security is based on a moment of trust that is granted by a user at some point in time – and is assumed thereafter (once given, hard to rescind). Cryptographic key methods only remove the need for a repeat of the trust decision. After the first exchange, trust is no longer needed, because the keys allow identity to be actually verified.
Even if you leave the trust options switched on, you are not blindly trusting
the hosts you know about. The only potential insecurity lies in any new keys
that you have not thought about. If you use wildcards or IP prefixes in the
trust rules, then other hosts might be able to spoof their way in on trust
because you have left open a hole for them to exploit. That is why it is
recommended to set the system to the state of zero trust
immediately after key transfer, by commenting or emptying out the trust options
(trustkeysfrom
on the server).
It is possible, though somewhat laborious, to transfer the keys out of band,
by copying /var/cfengine/ppkeys/localhost.pub
to
/var/cfengine/ppkeys/user-aaa.bbb.ccc.mmm
(assuming IPv4) on another host.
e.g.
localhost.pub -> root-128.39.74.71.pub
Other users than root
CFEngine normally runs as user "root" (except on Windows which does not normally have a root user), i.e. a privileged administrator. If other users are to be granted access to the system, they must also generate a key and go through the same process. In addition, the users must be added to the server configuration file.
Encryption
CFEngine provides encryption for keeping file contents private during transfer. If protocol_version
is set to classic
or 1
, then file transfers must be explicitly encrypted by setting encrypt=>"true"
in a copy_from
body of a files
promise. For newer protocol_version, all transfers are encrypted.
However, the main role of encryption in configuration management is for authentication. Secrets should not be transferred through policy, encrypted or not. Policy files should be considered public, and any leakage should not reveal secret information.
Troubleshooting
When setting up cf-serverd
, you might see the error message
Unspecified server refusal
This means that cf-serverd
is unable or is unwilling to authenticate the
connection from your client machine. The message is deliberately non-specific
so that anyone attempting to attack or exploit the service will not be given
information which might be useful to them.
There is a simple checklist for curing this problem:
- Make sure that you have granted access to the client's address in the
server control
body. - Make sure the connecting client is granted access to the requested resources
(files usually) in the
access_rules
promise bundle. - See the verbose log of the server for the exact error message, since the
client always gets the "Unspecified server refusal" reply from the server.
To run the server in verbose, kill cf-serverd on the policy hub and run:
$ cf-serverd -v
and then manually run
cf-agent
on the client. - In the unlikely case that you still get no indication of the denial, try
increasing the agent run verbosity.
cf-agent -I
for info-level messages or evencf-agent -v
for verbose.
Latest Release
New in CFEngine Learn about the newest features in CFEngine 3.6
Supported Platforms and Versions These are the supported platforms for the current release.
Policy Framework Updates 3.6 introduces significant changes to the masterfiles policy framework. Read more about these changes.
Known Issues View any issues of which we are currently aware and investigating. View possible workarounds.
New in CFEngine
CFEngine 3.6 is the latest version of CFEngine. The Enterprise edition introduces numerous improvements to the Mission Portal - a dashboard for getting a quick overview of the system status and compliance, alerts and notifications for changes to the system, a new user interface for inventory management and a tighter integration with Design Center.
Policy writers and community users, CFEngine 3.6 introduces a broad range of improvements and new functionality to the policy language, new capabilities for configuration management and tighter integration with data from external sources.
Improvements for Policy Writers
A number of changes have been made to the CFEngine core to further improve stability, scalability and security.
User Account Management
The new users
promise type allows for management and configuration of local user
accounts with CFEngine.
Data Containers for Structured Data
The new variable type data
allows handling of structured data within CFEngine policy.
Structured data can be read from JSON strings and files, which allows integrating of
CFEngine policies with any external data source that can export to JSON.
A range of new functions allow creating and operating on structured data.
Tagging with Meta Data
Classes and variables can be tagged with meta data for extended annotation and documentation of knowledge directly in the language. Classes and variables can be searched by tags in policy, and CFEngine Enterprise uses tags to identify relevant information reported by the host. CFEngine sets a number of tags on hard classes and special variables by default.
Improved File Templating Engine
CFEngine 3.6 introduces support for mustache
templates, which
is tightly integrated with data containers and provides easy and data-driven configuration
file management.
New and Improved Functions
CFEngine 3.6 adds a list of new built-in functions:
bundlesmatching()
- returns list of defined bundles matching a regexcanonifyuniquely()
- converts a string into a unique, legal class nameclassesmatching()
- returns list of set classes matching a regexeval()
- evaluates mathematical expressionsfindfiles()
- returns list of files matching a search patternmakerule()
- evaluates whether a target file needs to be rebuilt from sourcesmax()
,mean()
,min()
- returns maximum, mean and minimum in a container or listpackagesmatching()
- returns a filtered list of installed packages.string_downcase()
,string_upcase()
- returns the lower-/upper-case version of a stringstring_head()
,string_tail()
- returns the beginning/end of a stringstring_length()
- returns the length of a stringstring_reverse()
- reverses a stringvariablesmatching()
- returns a list of variables matching a regexvariance()
- returns the variance of numbers in a list
The following functions are added and improved to work with the new data
container
type:
data_readstringarray()
- read a delimited file into a data mapdata_readstringarrayidx()
- read a delimited file into a data arraydatastate()
- create adata
variable with currently set classes and variablesdatatype()
-format()
-%S
can be used to serialize 'data' containers into a stringmergedata()
- merge two data containersparsejson()
- create a data container from a JSON stringreadjson()
- create a data container from a file that contains JSONstorejson()
- serialize a data container into a string
Network Protocol on top of TLS
CFEngine 3.6 introduces a new networking protocol, which uses TLS for authentication. After the authentication, all dialog is encrypted within the established TLS session.
To ease with the upgrade process, cf-serverd
is still able to speak the legacy
protocol with old agents, and new agents can speak the legacy protocol with old servers.
CFEngine operators should turn off support for the legacy protocol as soon as all hosts
are running 3.6 to benefit from the full encryption and future improvements.
The new network protocol supports a range of new attributes for more precise access control to server resources.
Other Core Changes
CFEngine 3.6 uses LMDB as the default embedded database. LMDB is both robust and fast, and replaces TokyoCabinet on the majority of supported platforms.
Logging output of the CFEngine binaries has been further improved: cf-serverd
now includes
the relevant client IP address in all messages; reports
promises generate messages without
prefix except R:
, and execution state information (such as passes or promise type
changes) are included in the log.
Relative paths are supported in copy_from
promises, and can used together with the
shortcut
aliasing based on connection
data in access
promises to implement simple
and dynamic multi-tenancy within a single policy server.
CFEngine Enterprise
Mission Portal
CFEngine 3.6 introduces a flexible and dynamic dashboard with alerts and notifications to the Mission Portal. Users can create conditions and group those together in dashboard widgets that provide a quick overview over various business-critical aspects of the CFEngine managed system.
Inventory reports provide fast access to the managed assets, be it hosts or software. Inventory data can be added through policy to read additional information from sources available to each host, such as a CMDB.
Enterprise Platform and API
The Enterprise platform in 3.6 is using PostgreSQL, which significantly improves the speed of reports in the Mission Portal and through the Enterprise APIs. The Enterprise Server uses an optimized data collection protocol that reduces the network traffic and allows for even greater scalability.
The Enterprise API provides faster access to more data about hosts, promise compliance, software inventories, classes and variables. Thanks to the PostgreSQL backend the API supports complex SQL queries. The API from CFEngine Enterprise 2.2 is no longer supported.
The integration with authentication services (LDAP or ActiveDirectory) has changed to allow filter stings to be specified, providing greater flexibility in controlling access to CFEngine Enterprise.
Windows Support
A number of improvements to CFEngine's string handling on Windows makes sure that line endings are preserved in modified files, and use Windows compliance CRLF endings in newly created files.
Change History
For a complete history of changes in CFEngine, see the ChangeLog
and
Enterprise ChangeLog
.
ChangeLog
For a history of changes in the CFEngine version you have installed, see the ChangeLog
and
ChangeLog.Enterprise
files in /var/cfengine/share/doc
. The running change log
for the core repository is also available
online.
3.6.7: Changes: - classesmatching(): order of classes changed Fixes: - Don't limit CSV line length to 4096 bytes (Redmine #7400) - Legacy package promise: Result classes are now defined if the package being promised is already up to date. (Redmine #7399) - Reduce malloc() thread contention on heavily loaded cf-serverd, by not exiting early in the logging function, if no message is to be printed (Redmine #7624) - If file_select.file_types is set to symlink and there are regular files in the scanned directory, CFEngine no longer produces an unneccessary error message (Redmine #6996) - For call collect in Enterprise, default collect_window setting increased from 10 to 30 seconds for reliability reasons in large-scale environments. - Fix ps options for FreeBSD to check processes only in current host and not in jails - Fix HP-UX specific bug that caused a lot of log output to disappear - fix build failure on FreeBSD 7.1 (Redmine #7415) - Fix noise from internal policy to upgrade windows agents (Redmine #7456) - fix cfruncommand not working if it does not contain spaces, with the TLS protocol (Redmine #7405) - Fix cf-serverd error messages with classic protocol clients (Redmine #7818) - Fix two cases where action_policy warn still produces errors (Redmine #7274) - action_policy "warn" now correctly produces warnings instead of various other verbosity levels. (Redmine #7274) - No longer hang when changing permissions/ownership on fifos (Redmine #7030) - Fix a regression which would sometimes cause "Permission denied" errors on files inside directories with very restricted permissions. (Redmine #7808) - Fix classes being set because of hash collision in the implementation. (Redmine #7912) - redmine#7952: Fix use-after-free in ArrayMap and HashMap (Redmine #7952) - CFEngine on Windows no longer truncates log messages if the program in question is killed halfway through. - If there is an error saving a mustache template file it is now logged with log-level error (was inform) - Fix build with musl libc (Redmine #7455) - Show errors regarding failure to copy extended attributes when doing a local file copy. Errors could happen when copying across two different mount points where the support for extended attributes is different between the mount points. - The [`isvariable()`][isvariable] function call now correctly accepts all array variables when specified inline. Previously it would not accept certain special characters, even though they could be specified indirectly by using a variable to hold it (Redmine #7088) - Fix bad option nlwp to vzps on Proxmox / OpenVZ (Redmine #6961) - Namespaced classes can now be specified on the command line. - cf-agent, cf-execd, cf-promises, cf-runagent and cf-serverd honor multiple -D, -N and -s arguments (Redmine #7191) - Fix file descriptor leak when there are network errors - Fix package not installing on Windows 2008 32-bit (Redmine #7478) - Fixed several bugs which prevented CFEngine from loading libraries from the correct location. This affected several platforms. (Redmine #6708) - It is possible to edit the same value in multiple regions of one file (Redmine #7460) - Upgrade LMDB to version 0.9.17. (Redmine #7879) - Upgrade dependencies to following patch versions: - Apache 2.2.31 - apr-util 1.5.4 - Git 1.8.5.6 - Curl 7.47.0 - Libvirt 1.1.3.9 - LibXML 2.9.3 - LMDB 0.9.18 - MySQL 5.1.72 - OpenLDAP 2.4.44 - OpenSSL 0.9.8zh - PCRE 8.38 - PostgreSQL 9.0.22 (Agent) - PostgreSQL 9.3.11 (Hub) - PHP 5.4.45 3.6.6: Bug fixes: - Fixed insert_lines with preserve_block not converging if the block contains blank lines. (Redmine #7094) - Fixed incorrect message that permissions were repaired in dry-run mode. (Redmine #7082) - Fixed LMDB database sometimes getting corrupted on HP-UX, especially under load. (Redmine #6994) - Fixed and improve error messages in process promises. (Redmine #7219) - Fixed "cf-agent --self-diagnostics" segfaulting. (Redmine #7288) - Fixed a bug causing cf-agent to write at least 100MB to disk after every execution. (Redmine #7242) - Fixed a bug which caused cf-execd to react very slowly to signals. (Redmine #6659 and #7193) - Fixed a bug which could cause cf-execd to immediately kill itself after a reboot. (Redmine #7244) - Fixed systemd detection when /sbin/init is a symlink (Redmine #7297) - Removed useless error messages about failure to set socket timeout. The message can still be seen in verbose mode. (Redmine #6486) - Fixed a bug causing user creation not to work on HP-UX. (Redmine #6734) - Fixed broken bootstrap on Debian 8. Note that Debian 8 is not an officially supported platform with the 3.6 series. (Redmine #7042) - Fixed report collection not working correctly after initial install on systemd systems. (Redmine #7045) - Fixed a bug which caused service script to be unable to kill daemons on openvz/lxc hosts. (Redmine #3394) - Fixed replace_or_add bundle which would insert an extra line if used multiple times. (Redmine #7035) - "redhat_pure" hosts no longer includes Oracle Linux. (Redmine #7081) - Removed useless messages from "standard_services" bundle. (Redmine #7196) - Fixed a bug which causes "standard_services" bundle not to start a service unless it was already running. (Redmine #7258) - Fix process_result logic to match the purpose of body process_select days_older_than (Redmine #3009) 3.6.5: Features: - Introduced "systemd" hard class. (Redmine #6995) - Added paths to dtrace, zfs and zpool on FreeBSD in masterfiles. Bug fixes: - Fixed build error on certain RHEL5 and SLES10 setups. (Redmine #6841) - Fixed a bug which caused dangling symlinks not to be removed. (Redmine #6582) - Fixed data_readstringarrayidx function not preserving the order of the array it's producing. (Redmine #6920) - Fixed a bug which sometimes caused CFEngine to kill the wrong daemon if both the host and a container inside the host were running CFEngine. (Redmine #6906) - Made sure the rm_rf_depth bundle also deletes the base directory. (Redmine #7009) - Fixed monitord reporting wrongly on open ports. (Redmine #6926) - Skip adding the class when its name is longer than 1024 characters. Fixed core dump when the name is too large. (Redmine #7013) - Fixed detection of stopped process on Solaris. (Redmine #6946) - Fixed infinite loop (Redmine #6992) plus a couple more minor bugs in edit_xml promises. 3.6.4: Features: - Introduced users promises support on HP-UX platform. - Introduced process promises support on HP-UX platform. Bug fixes: - Fixed bug on FreeBSD which sometimes led to the wrong process being killed (Redmine #2330) - Fixed package version comparison sometimes failing with rpm package manager (Redmine #6807) - Fixed a bug in users promises which would sometimes set the wrong password hash if the user would also be unlocked at the same time. - Fixed a bug on AIX which would occationally kill the wrong process. - Improved error message for functions that require an absolute path. (Redmine #6877) - Fixed some spelling errors in examples. - Fixed error in out-of-tree builds when building cf-upgrade. - Fixed a bug which would make cf-agent exit with an error if it was built with a custom log directory, and that directory did not exist. - Fixed ordering of evaluating promises when depends_on is used. (Redmine #6484, Redmine #5462) - Skip non-empty directories silently when recursively deleting. (Redmine #6331) - Fix memory exhaustion with list larger than 4994 items. (Redmine # 6672) - Fix cf-execd segfault on IP address detection (Redmine #6905). - Fix hard class detection of RHEL6 ComputeNode (Redmine #3148). 3.6.3 New features: - support for HP-UX 11.23 and later - experimental support for Red Hat Enterprise Linux 7 Bug fixes: - fix getindices on multi-dimensional arrays (Redmine #6779) - fix mustache template method to run in dryrun mode (Redmine #6739) - set mailto and mailfrom settings for execd in def.cf (Redmine #6702) - fix conflation of multi-index entries in arrays (Redmine #6674) - fix promise locking when transferring using update.cf (Redmine #6623) - update JSON parser to return an error on truncation (Redmine #6608) - fix sys.hardware_addresses not expanded (Redmine #6603) - fix opening database txn /var/cfengine/cf_lastseen.lmdb: MDB_READERS_FULL when running cf-keys --show-hosts (Redmine #6602) - fix segfault (Null pointer dereference) when select_end in delete_lines never matches (Redmine #6589) - fix max_file_size => "0" not disabling or allowing any size (Redmine #6588) - fix ifvarclass, with iteration over list, failing when deleting files with time condition (Redmine #6577) - fix classes defined with "or" constraint are never set if any value doesn't evaluate to a scalar (Redmine #6569) - update "mailfrom" default in default policy (Redmine #6567) - fix logrotate ambiguity of filename (Redmine #6563) - fix parsing JSON files (Redmine #6549) - reduce write count activity to /var partition (Redmine #6523) - fix files delete attribute incorrectly triggering promise_kept (Redmine #6509) - update services bundle output related to chkconfig when run in inform mode. (Redmine #6492) - fix Solaris serverd tests (Redmine #6406) - fix broken bechaviour of merging arrays with readstringarray (Redmine #6369) - fix ifelapsed bug with bundle nesting (Redmine #6334) - fix handling cf_null in bundlesequence (Redmine #6119) - fix maparray reading whole input array when using subarray (Redmine #6033) - fix directories being randomly changed to files (Redmine #6027) - update defaults promise type to work with classes (Redmine #5748) - systemd integration in services promises (Redmine #5415) - fix touch attribute ignoring action = warn_only (Redmine #3172) - fix 4KB string limit in functions readfile, string_downcase, string_head, string_reverse, string_length, string_tail, string_upcase (Redmine #2912) 3.6.2 Bug fixes: - don't regenerate software_packages.csv every time (Redmine #6441) - improve verbose message for package_list_command - fix missing log output on AIX (Redmine #6434) - assorted fixes to dirname() esp on Windows (Redmine #4716) - fix package manager detection - fix build issues on FreeBSD - allow copying of dead symbolic links (Redmine #6175) - preserve order in readstringarrayidx (Redmine #6466) - fix passing of unexpanded variable references to arrays (Redmine #5893) - use entries for new {admin,deny}_{ips,hostnames} constraints in the relevant legacy lists (Redmine #6542) - cope with ps's numeric fields overflowing to the right - interpret failing function calls in ifvarclass as class not set (Redmine #6327) - remove unexpanded lists when extending lists (Redmine #6541) - infer start-time of a process from elapsed when needed (Redmine #4094) - fix input range definition for laterthan() function (Redmine #6530) - don't add trailing delimiter when join()'ing lists ending with a null-value (Redmine #6552) - 9999999999 (ten 9s) or higher has been historically used as an upper bound in CFEngine code and policy but because of overflow on 32-bit platforms it caused problems with big numbers. Fixed in two ways: first change all existing policy uses to 999999999 (nine 9s instead of eleven 9s), second fix the C code to not wrap-around in case of overflow, but use the LONG_MAX value (Redmine #6531). - cf-serverd and other daemons no longer reload their configuration every minute if CFEngine is built with an inputs directory outside of the work directory (not the default). (Redmine #6551) 3.6.1 New features: - Introduced Solaris and AIX support into the 3.6 series, with many associated build and bug fixes. Changes: - Short-circuit evaluation of classes promises if class is already set (Redmine #5241) - fix to assume all non-specified return codes are failed in commands promises (Redmine #5986) - cf-serverd logs reconfiguration message to NOTICE (was INFO) so that it's always logged in syslog Bug fixes: - File monitoring has been completely rewritten (changes attribute in files promise), which eliminates many bugs, particularly regarding files that are deleted. Upgrading will keep all monitoring data, but downgrading again will reinitialize the DB, so all files will be reported as if they were new. (Redmine #2917) - $(this.promiser) expands in files promises for 'transformer', 'edit_template', 'copy_from.source', 'file_select.exec_program', 'classes' and 'action' bodies (Redmine #1554, #1496, #3530, #1563) - 'body changes' notifies about disappeared files in file monitoring (Redmine #2917) - Set not-kept classes when files or commands promise should be repaired, but is warn-only (Redmine #2359) - Fixed CFEngine template producing a zero-sized file (Redmine #6088) - Add 0-9 A-Z _ to allowed context of module protocol (Redmine #6063) - Extend ps command column width on Solaris and filter on zone rather than adding it to the ps output. - Fixed strftime() function on Solaris when called with certain specifiers. - Fixed users promise bug regarding password hashes in a NIS/NSS setup. - Fixed $(sys.uptime), $(sys.systime) and $(sys.sysday) in AIX. (Redmine #5148, #5206) - Fixed processes_select complaining about "Unacceptable model uncertainty examining processes" (Redmine #6337) - ps command for linux has been changed to cope with big rss values (Redmine #6337) - Address ps -axo shift on FreeBSD 10 and later (Redmine #5667) - methods and services promises respect action_policy => "warn" (Redmine #5924) - LMDB should no longer deadlock if an agent is killed on the hub while holding the DB lock. Note that the change only affects binary packages shipped by CFEngine, since the upstream LMDB project has not yet integrated the change. (Redmine #6013) 3.6.0 Changes: - Changes to logging output - add process name and pid in syslog message (GitHub #789) - cf-serverd logging levels are now more standardised: - INFO logs only failures - VERBOSE logs successful requests as well - DEBUG logs actual protocol traffic. - cf-serverd now logs the relevant client IP address on each message. - Logging contexts to local database (cf_classes.tcdb) has been deprecated. - 'usebundle' promisees are logged for all the bundle promises - output from 'reports' promises has nothing prefixed except 'R: ' - a log line with stack path is generated when the promise type evaluated changes - LMDB (symas.com/mdb) is the default database for local data storage : use version 0.9.9 or later cf-agent --self-diagnostics (-x) is only implemented for TCDB, not for LMDB - port argument in readtcp() and selectservers() may be a service name (e.g. "http", "pop3"). - Enable source file in agent copy_from promises to be a relative path. - file "changes" reporting now reports with log level "notice", instead of "error". - process_results default to AND'ing of set attributes if not specified (Redmine #3224) - interface is now canonified in sys.hardware_mac[interface] to align with sys.ipv4[interface] (Redmine #3418) - cf-promises no longer errors on missing bodies when run without --full-check (-c) - Linux flavor "SUSE" now correctly spelled with all uppercase in variables and class names (Redmine #3734). The "suse" lowercase version is also provided for convenience (Redmine #5417). - $(this.promise_filename) and $(..._dirname) variables are now absolute paths. (Redmine #3839) - including the same file multiple times in 'body control inputs' is not an error - portnumber in body copy_from now supports service names like "cfengine", "pop3" etc, check /etc/services for more. - The failsafe.cf policy, run on bootstrap and in some other unusual cases, has been extracted from C code into libpromises/failsafe.cf - masterfiles - cf_promises_validated is now in JSON format - timestamp key is timestamp (sec since unix epoch) of last time validated - the masterfiles now come from https://github.com/cfengine/masterfiles and are not in the core repository - cf-serverd calls cf-agent with -Dcfruncommand when executing cf-runagent requests - Mark as removed: promise_notkept_log_include, promise_notkept_log_exclude, promise_repaired_log_include, promise_repaired_log_exclude, classes_include, classes_exclude, variables_include, variables_exclude attributes from report_data_select body (syntax is valid but not functional). They have been replaced by the following attributes: promise_handle_include, promise_handle_exclude, metatags_include, metatags_exclude. New features: - New promise type "users" for managing local user accounts. - TLS authentication and fully encrypted network protocol. Additions specific to the new type of connections: - New attribute "allowlegacyconnects" in body server control, which enables serving policy via non-latest cfengine protocol, to the given list of hosts. If the option is absent, it defaults to allow all hosts. To refuse non-TLS connections, specify an empty list. - New attribute "protocol_version" in body copy_from, and body common control, which defines the preferred protocol for outgoing connections.. Allowed values at the moment: "0" or "undefined", "classic" or "1", "latest" or "2". By leaving the copy_from option as undefined the common control option is used, and if both are undefined then classic protocol is used by default. - The new networking protocol uses TLS for authentication, after which all dialog is encrypted within the established TLS session. cf-serverd is still able to speak the legacy protocol with old agents. - The 'skipverify' option in 'body server control' is deprecated and only left for compatibility; it does nothing - cf-serverd does not hang up the connection if some request fails, so that the client can add more requests. - For the connections using the new protocol, all of the paths in bundle server access_rules now differentiate between a directory and a file using the trailing slash. If the path exists then this is auto-detected and trailing slash appended automatically. You have to append a trailing slash manually to an inexistent or symbolic path (e.g. "/path/to/$(connection.ip)/") to force recursive access. - New in 'access' promises for 'bundle server access_rules' - Attributes "admit_ips", "admit_hostnames", "admit_keys", "deny_ips", "deny_hostnames", "deny_keys" - "admit_keys" and "deny_keys" add the new functionality of controlling access according to host identity, regardless of the connecting IP. - For these new attributes, regular expressions are not allowed, only CIDR notation for "admit/deny_ips", exact "SHA=..." strings for "admit/deny_keys", and exact hostnames (e.g. "cfengine.com") or subdomains (starting with dot, e.g. ".cfengine.com") for "admit/deny"_hostnames. Same rules apply to 'deny_*' attributes. - These new constraints and the paths in access_rules, can contain special variables "$(connection.ip)", "$(connection.hostname)", "$(connection.key)", which are expanded dynamically for every received connection. - For connections using the new protocol, "admit" and "deny" constraints in bundle server access_rules are being phased out, preferred attributes are now "admit_ips", "deny_ips", "admit_hostnames", "deny_hostnames", "admit_keys", "deny_keys". - New "shortcut" attribute in bundle server access_rules used to dynamically expand non-absolute request paths. - masterfiles - standard library split: lib/3.5 (compatibility) and lib/3.6 (mainline) - many standard library bundles and bodies, especially packages- and file-related, were revised and fixed - supports both Community and Enterprise - new 'inventory/' structure to provide OS, dmidecode, LSB, etc. system inventory (configured mainly in def.cf) - cf_promises_release_id contains the policy release ID which is the GIT HEAD SHA if available or hash of tree - a bunch'o'bundles to make starting with CFEngine easier: - file-related: file_mustache, file_mustache_jsonstring, file_tidy, dir_sync, file_copy, file_link, file_hardlink, file_empty, file_make - packages-related: package_absent, package_present, package_latest, package_specific_present, package_specific_absent, package_specific_latest, package_specific - XML-related: xml_insert_tree_nopath, xml_insert_tree, xml_set_value, xml_set_attribute - VCS-related: git_init, git_add, git_checkout, git_checkout_new_branch, git_clean, git_stash, git_stash_and_clean, git_commit, git - process-related: process_kill - other: cmerge, url_ping, logrotate, prunedir - New command line options for agent binaries - New options to cf-promises - '--show-classes' and '--show-vars' - '--eval-functions' controls whether cf-promises should evaluate functions - Colorized output for agent binaries with command line option '--color' (auto-enabled if you set CFENGINE_COLOR=1) - New language features - New variable type 'data' for handling of structured data (ie JSON), including supporting functions: - 'data_readstringarray' - read a delimited file into a data map - 'data_readstringarrayidx' - read a delimited file into a data array - 'datastate' - create a data variable with currently set classes and variables - 'datatype' - determine the type of the top element of a container - 'format' - %S can be used to serialize 'data' containers into a string - 'mergedata' - merge two data containers, slists/ilists/rlists, or "classic" arrays into a data container - 'parsejson' - create a data container from a JSON string - 'readjson' - create a data container from a file that contains JSON - 'storejson' - serialize a data container into a string - Most functions operating on lists can also operate on data containers - pass a data container to a bundle with the @(container) notation - the module protocol accepts JSON for data containers with the '%' sigil - Tagging of classes and variables allows annotating of language construct with meta data; supporting functionality: - The module protocol in 'commands' promises has been extended to allow setting of tags of created variables and classes, and the context of created variables - 'getclassmetatags' - returns list of meta tags for a class - 'getvariablemetatags' - returns list of meta tags for a variable - 'body file control' has an 'inputs' attribute to include library files and other dependencies - bundlesequences can be built with bundlesmatching() based on bundle name and tags - New attributes in existing promise types and bodies - New option 'preserve_all_lines' for insert_type in insert_lines promises - Caching of expensive system functions to avoid multiple executions of execresult() etc, can be controlled via cache_system_functions attribute in body common control - New option 'mailsubject' in body executor control allows defining the subject in emails sent by CFEngine - Support for Mustache templates in 'files' promises; use 'template_method' and 'template_data' attributes. Without 'template_data' specified, uses datastate(). - New and improved functions - 'bundlesmatching' - returns list of defined bundles matching a regex and tags - 'canonifyuniquely' - converts a string into a unique, legal class name - 'classesmatching' - returns list of set classes matching a regex and tags - 'eval' - evaluates mathematical expressions; knows SI k, m, g quantifiers, e.g. "100k" - 'findfiles' - list files matching a search pattern; use "**" for recursive searches - 'makerule' - evaluates whether a target file needs to be rebuilt from sources - 'max', 'min' - returns maximum and minimum of the numbers in a container or list (sorted by a 'sort' method) - 'mean' - returns the mean of the numbers in a container or list - 'nth' - learned to look up by key in a data container holding a map - 'packagesmatching' - returns a filtered list of installed packages. - 'readfile' - learned to read system files of unknown size like those in /proc - 'sort' - can sort lexicographically, numerically (int or real), by IP, or by MAC - 'string_downcase', 'string_upcase' - returns the lower-/upper-case version of a string - 'string_head', 'string_tail' - returns the beginning/end of a string - 'string_length' - returns the length of a string - 'string_reverse' - reverses a string - 'string_split' - improved implementation, deprecates 'splitstring' - 'variablesmatching' - returns a list of variables matching a regex and tags - 'variance' - returns the variance of numbers in a list or container - New hard classes - Introduced alias 'policy_server' for context 'am_policy_hub' (the latter will be deprecated) - all the time-based classes have GMT equivalents - New variables - 'sys.bindir' - the location of the CFEngine binaries - 'sys.failsafe_policy_path' - the location of the failsafe policy file - 'sys.inputdir' - the directory where CFEngine searches for policy files - 'sys.key_digest' - the digest of the host's cryptographic key - 'sys.libdir', 'sys.local_libdir' - the location of the CFEngine libraries - 'sys.logdir' - the directory where the CFEngine log files are saved - 'sys.masterdir' - the location of masterfiles on the policy server - 'sys.piddir' - the directory where the daemon pid files are saved - 'sys.sysday' - the number of days since the beginning of the UNIX epoch - 'sys.systime' - the number of seconds since the beginning of the UNIX epoch - 'sys.update_policy_path' - the name of the update policy file - 'sys.uptime' - the number of minutes the host has been online - 'this.promise_dirname' - the name of the file in which the current promise is defined - 'this.promiser_uid' - the ID of the user running cf-agent - 'this.promiser_gid' - the group ID of the user running cf-agent - 'this.promiser_ppid' - the ID of the parent process running cf-agent Deprecations: - 'splitstring' - deprecated by 'string_split' - 'track_value' - 'skipverify' Bug fixes: for a complete list of fixed bugs, see Redmine at https://cfengine.com/dev - various fixes in evaluation and variable resolution - Improve performance of list iteration (Redmine #1875) - Removed limitation of input length to internal buffer sizes - directories ending with "/" are not ignored - lsdir() always return a list now, never a scalar - 'abortclasses' fixed to work in common bundles and other cases - namespaced 'edit_line' bundles now work (Redmine#3781) - lists are interpolated in correct order (Redmine#3122) - cf-serverd reloads policies properly when they change - lots of leaks (memory and file descriptor) fixed 3.5.3 Changes: - Improved security checks of symlink ownership. A symlink created by a user pointing to resources owned by a different user will no longer be followed. - Changed the way package versions are compared in package promises. (Redmine #3314) In previous versions the comparison was inconsistent. This has been fixed, but may also lead to behavior changes in certain cases. In CFEngine 3.5.3, the comparison works as follows:For instance: apache-2.2.31 ">=" "2.2.0" will result in the package being installed. Bug fixes: - fix cf-monitord crash due to incorrect array initialization (Redmine #3180) - fix cf-serverd stat()'ing the file tree every second (Redmine #3479) - correctly populate sys.hardware_addresses variable (Redmine #2936) - add support for Debian's GNU/kfreebsd to build system (Redmine #3500) - fix possible stack corruption in guest_environments promises (Redmine #3552) - work-around hostname trunctation in HP-UX's uname (Redmine #3517) - fix body copy purging of empty directories (Redmine #3429) - make discovery and loading of avahi libraries more robust - compile and packaging fixes for HP-UX, AIX and Solaris - fix fatal error in lsdir() when directory doesn't exist (Redmine #3273) - fix epoch calculation for stime inrange calculation (Redmine #2921) 3.5.2 Bug fixes: - fix delayed abortclasses checking (Redmine #2316, #3114, #3003) - fix maplist arguments bug (Redmine #3256) - fix segfaults in cf-pomises (Redmine #3173, 3194) - fix build on Solaris 10/SmartOS (Redmine #3097) - sanitize characters from /etc/issue in sys.flavor for Debian (Redmine #2988) - Fix segfault when dealing with files or data > 4K (Redmine #2912, 2698) - Don't truncate keys to 126 characters in getindices (Redmine #2626) - files created via log_* actions now have mode 600 (Redmine #1578) - fix wrong log message when a promise is ignored due to 'ifvarclass' not matching - fix lifetime of persistent classes (Redmine #3259) - fix segfault when process_select body had no process_result attribute Default to AND'ed expression of all specified attributes (Redmine #3224) - include system message in output when acl promises fail - fix invocation of standard_services bundle and corresponding promise compliance (Redmine #2869) 3.5.1 Changes: - file changes are logged with log level Notice, not Error - the CFEngine Standard Library in masterfiles/libraries is now split into promise-type specific policy files, and lives in a version-specific directory. This should have no impact on current code, but allows more granular include of needed stdlib elements (Redmine #3044) Bug fixes: - fix recursive copying of files (Redmine #2965) - respect classes in templates (Redmine ##2928) - fix timestamps on Windows (Redmine #2933) - fix non-root cf-agent flooding syslog (Redmine #2980) - fix email flood from cf-execd due to timestamps in agent output (Redmine #3011) - Preserve security context when editing or copying local files (Redmine #2728) - fix path for sys.crontab on redhat systems (Redmine #2553) - prevent incorrect "insert_lines promise uses the same select_line_matching anchor" warning (Redmine #2778) - Fix regression of setting VIPADDRESS to 127.0.0.1 (Redmine #3010) - Fix "changes" promise not receiving status when file is missing (Redmine #2820) - Fix symlinks being destroyed when editing them (Redmine #2363) - Fix missing "promise kept" status for the last line in a file (Redmine #2943) 3.5.0 New features: - classes promises now take an optional scope constraint. - new built-in functions: every, none, some, nth, sublist, uniq, filter - every - none - some - nth - sublist - uniq - filter - classesmatching - strftime - filestat - ifelse - maparray - format - cf-promises flag --parse-tree is replaced by --policy-output-format=, requiring the user to specify the output format (none, cf, json) - cf-promises allows partial check of policy (without body common control) without integrity check; --full-check enforces integrity check - agent binaries support JSON input format (.json file as generated by cf-promises) - cf-key: new options --trust-key/-t and --print-digest/-p - Class "failsafe_fallback" is defined in failsafe.cf when main policy contains errors and failsafe is run because of this - add scope attribute for body classes (Redmine #2013) - Better diagnostics of parsing errors - Error messages from parser now show the context of error - new cf-agent option: --self-diagnostics - new output format, and --legacy-output - warnings for cf-promises. - Enable zeroconf-discovery of policy hubs for automatic bootstrapping if Avahi is present - Support for sys.cpus on more platforms than Linux & HPUX Changes: - parser no longer allows ',' after promiser or promisee. must be either ';' or lval - Make parser output in GCC compatible format the only supported format (remove --gcc-brief-format flag) - Silence license warnings in Enterprise Free25 installations - action_policy => "warn" causes not_kept classes to be set on promise needing repair. - command line option version (-V) now prints a shorter parsable version without graphic - implicit execution of server and common bundles taking arguments is skipped in cf-serverd. - WARNING: option --policy-server removed, require option to --bootstrap instead - process promises don't log if processes are out of range unless you run in verbose mode - reports promises are now allowed in any context (Redmine #2005) - cf-report has been removed - cf-execd: --once implies --no-fork - Version info removed from mail subject in the emails sent by cf-execd. The subject will only contain "[fqname/ipaddress]" instead of "communnity/nova [fqname/ipaddress]" Please change your email filters accordingly if necessary. - "outputs" promise type is retired. Their semantics was not clear, and the functionality is better suited for control body setting, not a promise. - Tokyo Cabinet databases are now automatically checked for correctness during opening. It should prevent a number of issues with corrupted TC databases causing binaries to hang. - Improved ACL handling on Windows, which led to some syntax changes. We now consistently use the term "default" to describe ACLs that can be inherited by child objects. These keywords have received new names: acl_directory_inherit -> acl_default specify_inherit_aces -> specify_default_aces The old keywords are deprecated, but still valid. In addition, a new keyword "acl_inherit" controls inheritance behavior on Windows. This feature does not exist on Unix platforms. (Redmine #1832) - Networking code is moved from libpromises to its own library, libcfnet. Work has begun on making the API more sane and thread-safe. Lots of legacy code was removed. - Add getaddrinfo() replacement in libcompat (borrowed from PostgreSQL). - Replace old deprecated and non thread-safe resolver calls with getaddrinfo() and getnameinfo(). - Hostname2IPString(), IPString2Hostname() are now thread-safe, and are returning error when resolution fails. - Running cf-execd --once now implies --no-fork, and also does not wait for splaytime to pass. - execresult(), returnszero() and commands promises no longer requires the first word word to be an absolute path when using the shell. (Part of Redmine #2143) - commands promises useshell attribute now accepts "noshell" and "useshell" values. Boolean values are accepted but deprecated. (Part of Redmine #2143) - returnszero() now correctly sets the class name in this scenario (Part of Redmine #2143): classes: "commandfailed" not => returnszero("/bin/nosuchcommand", "noshell"); Bugfixes: - bundles are allowed to be empty (Redmine #2411) - Fixed '.' and '-' not being accepted by a commands module. (Redmine #2384) - Correct parsing of list variables by a command module. (Redmine #2239) - Fixed issue with package management and warn. (Redmine #1831) - Fixed JSON crash. (Redmine #2151) - Improved error checking when using fgets(). (Redmine #2451) - Fixed error message when deleting nonexistent files. (Redmine #2448) - Honor warn-only when purging from local directory. (Redmine #2162) - Make sure "restart" and "reload" are recognized keywords in packages. (Redmine #2468) - Allocate memory dynamically to avoid out-of-buffer or out-of-hash situations - fix edit_xml update of existing attributes (Redmine #2034) - use failsafe policy from compile-time specified workdir (Redmine #1991) - ifvarclass checked from classes promises in common bundles - do not wait for splaytime when executing only once - disable xml editing functionality when libxml2 doesn't provide necessary APIs (Redmine #1937) - Out-of-tree builds should work again, fixed a bunch of related bugs. - Fixed race condition in file editing. (Redmine #2545) - Fixed memory leak in cf-serverd and others (Redmine #1758) 3.4.5 (Bugfix and Stability release) Bugfixes: - make qualified arrays expand correcty (Redmine #1998, Mantis #1128) - correct possible errors in tcdb files when opening - avoid possible db corruption when mixing read/write and cursor operations - Allow umask value of 002 (Redmine #2496) 3.4.4 (Bugfix and Stability release) Bugfixes: - prevent possible crash when archiving files (GitHub #316) - don't create symlinks to cf-know in update policy - don't enable xml support if libxml2 is too old (Redmine #1937) 3.4.3 (Bugfix and Stability release) Bugfixes: - Don't flood error messages when processes are out of defined range - prevent segmentation fault in cf-monitord -x (Redmine #2021) - when copying files, use same file mode as source file, rather than 0600 (Redmine #1804) - include xpath in messages generated by edit_xml operations (Redmine #2057) 3.4.2 (Bugfix and Stability release) Bugfixes: - Fixes to policies in masterfiles (see masterfiles/Changelog for details) - Fixes for OpenBSD (GitHub #278) - Do not canonify values specified in abortbundleclasses/abortclasses (Redmine #1786) - Fix build issues on NetBSD, SLES 12.2 - Improve error message when libxml2 support is not compiled (Redmine #1799) - fix potential segmentation fault when trimming network socket data (GitHub #233) - fix potential segmentation fault when address-lookups in lastseen db failed (GitHub #233) - execute background promise serially when max_children was reached, rather than skipping them (GitHub #233) - fix segmentation fault in cf-promises when invoked with --reports (Redmine #1931) - fix compilation with Sun Studio 12 (Redmine #1901) - silence type-pun warning when building on HP-UX (GitHub #287) 3.4.1 (Bugfix and Stability release) New feature/behavior: - cf-execd terminates agent processes that are not responsive for a configurable amount of time (see agent_expireafter in body executor control), defaulting to 1 week Bugfixes: - fix regression of classmatch() failing with hard classes (Redmine #1834) - create promise-defined and persistent classes in correct namespace (Redmine #1836) - several fixes to namespace support - fix several crash bugs caused by buffer overflow and race conditions in cf-serverd - regenerate time classes in cf-execd for each run (Redmine #1838) - edit_xml: fix select_xpath implementation and update documentation NOTE: code that uses select_xpath_region needs to be changed to select_xpath - edit_xml: make sure that text-modification functions don't overwrite child nodes - edit_xml: improve error logging 3.4.0 New features: - Added rpmvercmp utility to compare versions of RPM packages for accurate sorting of RPM packages for packages promises. - Implement network timeout on server side to avoid keeping stale connections for hours. - XML editing capabilities. See the documentation for edit_xml body. Note the new dependency: libxml2. - Implement inheritance of local classes by bundles called using "usebundle". By default classes are not inherited. See the examples/unit_inherit.cf for an example. - Moved from Nova/Enterprise: - POSIX ACL support, - "outputs" promise type, - remote syslog support. - packages_default_arch_command hook in packages promises, to specify default architecture of the packages on the system. - packages_version_less_command / packages_version_equal_command hooks in packages promises, to specify external command for native package manager versions comparison - agent_expireafter in body executor control allows you to set a timeout on all cf-agent runs, to enforce a threshold on the number of concurrent agents - Running in Solaris zone is now detected and classes "zone" and "zone_ " are created in this case. - VirtualBox support added to guest_environment promises. - guest_environment promises are supported under OS X. - The "depends_on" attribute is now active, for the partal ordering of promises. If a promise depends on another (referred by handle) it will only be considered if the depends_on list is either kept or repaired already. ** WARNING: When upgrading, make sure that any existing use of depends_on does not make some promises being unintentionally ignored. This can happen if you are currently referring to non-existent or never-run handles in depends_on attributes. - methods return values, initial implementation - New format for cf-key -s, includes timestamp of last connection - cf-promises --parse-tree option to parse policy file and dump it in JSON format - Namespaces support for bundles and bodies. See the examples/unit_namespace*.cf for the usage. - Default arguments for bundles. See the examples/unit_defaults.cf - Metadata promise type. See the examples/unit_meta.cf New semantics: - Methods promises now return the status of promises kept within them. If any promise was not kept, the method is not kept, else if any promise is repaired, the method was repaired else it was kept. - Remote variable access in namespaces by $(namespace:bundle.variable) Changed functionality: - cf-execd -F switch no longer implies 'run once'. New -O/--once option is added to achieve this behaviour. This makes cf-execd easier to run from systemd, launchd and other supervision systems. Misc: - Support for the following outdated platforms and corresponding classes has been removed. De facto those platforms were unsupported for a long time, as CFEngine codebase uses C99 language features unavailable on old platforms: - SunOS 3.x (sun3) - SunOS 4.x (sun4) - Ultrix (ultrix) - DEC OSF/1 AXP (osf) - Digital UNIX (digital) - Sony NEWS (newsos) - 4.3BSD (bsd4_3) - IRIX (irix, irix4, irix64) - IBM Academic Operating System (aos) - BSD/OS / BSDi / BSD/386 (bsdos) - NeXTSTEP (nextstep) - GNU Hurd (gnu) - NEC UX/4800 (ux4800) - (Old news) Since 3.3.0 the layout of CFEngine Community packages has changed slightly. cf-* binaries have been moved to /var/cfengine/bin, due to the following reasons: - cf-* binaries are linked to libraries installed to /var/cfengine/lib, so placing binaries in /usr/local/sbin does not increase reliability of the CFEngine, - keeping whole CFEngine under single prefix (/var/cfengine) makes packaging simpler, - it matches the layout of CFEngine Enterprise packages. Please adjust your policies (the recommended ways to deal with the move are either to adjust $PATH to include /var/cfengine or to create symlinks in /usr/local/sbin in case you are relying on binaries to be available in $PATH). - Workdir location is properly changed if --prefix or --enable-fhs options are supplied to configure (Mantis #1195). - Added check for broken libmysqlclient implementations (Mantis #1217). - Standard library is updated from COPBL repository. - cf-know is no longer built in Community releases. The only functionality useful in Community, namely the reference manual generation, is provided by new compile-time cf-gendoc tool. - Filename (for storing filechanges) changed from file_change.log -> file_changes.log (in /var/cfengine/state) New format for storing file changes introduced: [timestamp,filename, ,Message] N = New file found C = Content Changed S = Stats changed R = File removed - Acceptance test suite passes on Mac OS X. - Changed some port numbers to replace old services with imap(s) - archlinux hard class on Arch Linux. - Detect BSD Make and automatically switch to GNU Make during build. Bugfixes: - cfruncommand for cf-execd is an arbitrary shell command now (Mantis #1268). - Fixed broken "daily" splayclasses (Mantis #1307). - Allow filenames up to 4096 bytes in network transfers (Redmine #1199). - Fix stale state preserved during cf-serverd reload (Redmine #1487). - Free disk space calculation is fixed (Mantis #1120). - Numerous portability bugfixes (especially OpenBSD, Solaris, AIX-related). - Compatibility fixes for AIX, HP-UX, Solaris (Mantis #1185, Mantis #1177, Mantis #1109). - Fixed broken socklen_t configure check under OpenBSD (Mantis #1168). - Fixed hang in cf-promises under OpenBSD (Mantis #1113). - Fixed endless loop in evaluating "$()" construct (Mantis #1023). - Fixed check for old PCRE versions (Mantis #1262). - Fixed insertion of multi-line blocks at the start of file (Mantis #809). - Fixed numerous memory leaks. - Fixes for metadata that were not resolvable - Fixes for namespaces that would not support metadata and variable expansion - Point-to-point network interfaces are detected and reported by CFEngine (Mantis #1246) - Partial non-GNU userspace support in acceptance testsuite (Mantis #1255) Full list of issues fixed is available on https://cfengine.com/bugtracker/changelog_page.php (old bug tracker) and https://cfengine.com/dev/projects/core/versions/34 (new bug tracker) 3.3.9 (Bugfix and Stability release) Bugfixes: - Do not lose hard classes in cf-serverd during policy reload (Mantis #1218). - Implement receive network timeout in cf-serverd. Prevents overloading cf-serverd with stale connections. 3.3.8 (Bugfix and Stability release) Versions 3.3.6, 3.3.7 were internal and weren't released. Bugfixes: - Propery set sys.domain variable if hostname is fully-qualified. - Fixed several small memory leaks. - Make network timeout for network reads configurable. Previously it was hardcoded to be 30 seconds, which was not enough for cf-runagent invoking cf-agent on big policies (Mantis #1028). 3.3.5 (Bugfix and Stability release) Bugfixes: - Fixed cf-execd memory leak on hosts with cf-monitord running. - Robustify against wrongly-sized entires in embedded databases. Standard library: - Bugfixes from upstream COPBL repository. - standard_services bundle from upstream COPBL repository. 3.3.4 (Bugfix and Stability release) Evaluation of policies: - Fix wrong classes set after installation of several packages using packages promises (Mantis #829). - Fix segfault using edit_template on existing file (Mantis #1155). Misc: - Fix memory leak during re-read of network interfaces' information in cf-execd/cf-serverd. 3.3.3 (Bugfix and Stability release) Evaluation of policies: - Zero-length files are valid for readfile() and similar functions (Mantis #1136). - Unchoke agent in case it encounters symlinks in form ./foo (Similar to Mantis #1117). Misc: - Fix generation of reference manual on machines with umask more relaxed than 022. - Use statvfs(3) on OpenBSD to obtain filesystem information (Mantis #1135). 3.3.2 (Bugfix and Stability release) Evaluation of policies: - Do not segfault if file copy was interrupted due to network connectivity or server going away (Mantis #1089). - Do not segfault if log_failed attribute is present in body, but log_kept is not (Mantis #1107). - Do not mangle relative paths in symlinks during file copy Previously symlink a -> b was mangled to a -> ./b. (Mantis #1117) - Properly compare 1.0 and 1.0.1 in packages promises. Previously only versions with equal amount of "segments" were comparable (Mantis #890, #1066). Base policy: - Properly set permissions on files for /var/cfengine/lib on HP-UX (Mantis #1114). - Standard library (cfengine_stdlib.cf) is synced with COPBL repository. Misc: - Do not create huge file in case corrupted TokyoCabinet database is detected (Mantis #1106). - Fix file descriptor leak on error paths, may have caused crashes of cf-execd and cf-serverd (Issue #1096). - Fix intermittent segfault in cf-execd (Mantis #1116). - Impose an upper limit on amount of listening sockets reported by cf-monitord. Huge amounts of listening sockets caused cf-agent to segfault on next run (Mantis #1098). - Add missing function prototypes caused errors during compilation on HP-UX (Mantis #1109). - Fix compilation on Solaris 11 (Mantis #1091). 3.3.1 (Bugfix and Stability release) Evaluation of policies: - Do not cut off name of bundle in variables interpolation (Mantis #975). - Do not segfault in function evaluation guarded by ifvaclass clause (Mantis #1084, #864). - Do not segfault if "classes" promise does not declare any value to be evaluated (Mantis #1074). - Do not segfault in database promises if there is no database_operation provided (Mantis #1046). Built-in functions: - Fix countclassesmatching() function which was misbehaving trying to match classes starting with alphanumeric symbol (Mantis #1073). - Fix diskfree() to return kilobytes, as described in documentation (Mantis #980, #955). - Fix hostsseen() function to avoid treating all hosts as not being seen since 1970 (Mantis #886). - Do not output misleading error message if readtcp() is unable to connect (Mantis #1085). Command-line interface: - -d option previously reqired an argument, though help message disagreed (Mantis #1053). - Disable --parse-tree option, not ready for the release (Mantis #1063). - Acept -h as a --help option. - Ensure that cf-execd might be started right after being shut down. Misc: - Plug file descriptor leak after failed file copy (Mantis #990). - Fix unsafe admit rules in default promises.cf (Mantis #1040). - Fix splaytime to match documentation: it is specified in minutes, not seconds (Mantis #1099). Packaging: - Fix owner/group of initscript and profile.d snippet in RPM builds (Mantis #1061, #1058). - Fix location of libvirt socket CFEngine uses to connect to libvirtd (Mantis #1072). - Install CoreBase to /var/cfengine/masterfiles during installation (Mantis #1075). - Do not leave old cf-twin around after upgrade (Mantis #1068) - Do not leave rcS.d symlinks after purging .deb package (Mantis #1092). 3.3.0 New promise types: - Guest environments promises, which allow to manipulate virtual machines using libvirt. - Database promises, which allow to maintain schema of MySQL and PostgreSQL databases. Database promises are in "technical preview" status: this promise type is subject to change in future. - Services promises for Unix, allows abstraction of details on managing any service New built-in functions: - dirname() to complement lastnode() - lsdir() - maplist() to apply functions over lists New features: - Allow defining arrays from modules. - Allow both `process_stop' and `signals' constraints in `processes' promises at the same time. - cf-promises --gcc-brief-format option to output warnings and errors in gcc-compatible syntax which to ease use "go to next error" feature of text editors. - Iteration over lists is now allowed for qualified (non-local) lists. New built-in variables and classes (Linux): - Number of CPUs: $(sys.cpus), 1_cpu, 2_cpus etc New built-in variables and classes (Unices): - $(sys.last_policy_update) - timestamp when last policy change was seen by host - $(sys.hardware_addresses) - list of MAC adresses - $(sys.ip_addresses) - list of IP addresses - $(sys.interfaces) - list of network interfaces - $(sys.hardware_mac[$iface]) - MAC address for network interface - mac_ :: - discovered MAC addresses Changes: - Major cleanup of database handling code. Should radically decrease amount of database issues experienced under heavy load. *WARNING*: Berkeley DB and SQLite backends are *removed*, use Tokyo Cabinet or QDBM instead. Both Tokyo Cabinet and QDBM are faster than Berkeley DB in typical CFEngine workloads. Tokyo Cabinet requires C99 environment, so it should be available on every contemporary operating system. For the older systems QDBM, which relies only on C89, is a better replacement, and deemed to be as portable, as Berkeley DB. - Change of lastseen database schema. Should radically decrease I/O contention on lasteen database. - Automatic reload of policies by cf-execd. - Documentation is generated during build, PDF and HTML files are retired from repository. - Rarely used feature retired: peer connectivity intermittency calculation. - Memory and CPU usage improvements. - Testsuite now uses 'make check' convention and does not need root privileges anymore. - cf_promises_validated now filled with timestamp, allows digest-copy for policy instead of mtime copy which is safer when clocks are unsynchronised - The bundled failsafe.cf policy now has trustkey=false to avoid IP spoofing attacks in default policy - See the full list of bugfixes at https://cfengine.com/bugtracker/changelog_page.php 3.2.4 (Bugfix and Stability release) Fixed failure in network transfer in case of misbehaving peer A few tiny memory leaks on error paths fixed 3.2.3 (Bugfix and Stability release) A few tiny memory leaks fixed Improved performance of cf-serverd under heavy load with TokyoCabinet database Full list of issues fixed is available on https://cfengine.com/bugtracker/changelog_page.php 3.2.2 (Bugfix and Stability release) Enabled compilation in "large files" mode under AIX Alleviated problem with broken file transfers over unstable Internet links. Full list of issues fixed is available on https://cfengine.com/bugtracker/changelog_page.php 3.2.1 (Bugfix and Stability release) Fixed compilation under HP-UX and Solaris Enabled compilation using HP ANSI C compiler Full list of issues fixed is available on https://cfengine.com/bugtracker/changelog_page.php 3.2.0 New bootstrap method with single-command bootstrapping: - cf-agent --bootstrap --policy-server 123.456.789.123 - Associated policy template files are added, partially maintained by CFEngine Bug fixes for file-editing, package versioning, and embedded database corruption (We recommend using TokyoCabinet instead of BerkeleyDB if building from source). Improved upgrade path for Nova. Patches for improved run-agent concurrency Reorganization of documentation and community resources 100% on regression test suite on 3 operating systems (Ubuntu, Debian, SuSE on x86-64 hardware) Support for multiple release environments package_policy update and addupdate now check if user-supplied version is larger than currently installed - updates only if so Help text of cf-report -r corrected - a list of key hashes is required, not ip addresses. New Emacs mode for CFEngine policy files (thanks to Ted Zlatanov!) Warnings are on edit_line changes can now give greater degree of information without spamming promise logs Class expressions parser accepts '||' as an alias for '|' again. Invalidation of package list cache on installation/removal of packages. New option cf-key -r to remove host key by IP or hostname. Added detection of network interfaces which belong to BSD jails. Improve robustness of multi-threaded code, in particular fix problems with spurious access denials in server and losing of authentication rules after policy reload. cf-promises accepts option -b matching cf-agent, which causes it to do not complain about missing bundlesequence. New functions and(), not(), or() and concat() to ease use of ifvarclass() clause. Full list of issues fixed is available on https://cfengine.com/bugtracker/changelog_page.php 3.1.5 New class parser, '||' is no longer allowed in expressions (use '|'). Class setting in the promise types insert_lines, delete_lines, replace_patterns, field_edits, vars, classes is restored. suspiciousnames implemented. New function getvalues(). New functions parse{read,int,string}array to match read{read,int,string}array. Testsuite added to check for core functionality. Syslog prefix is fixed to say 'cf3' instead of 'community'. 3.1.4 (Bugfix and Stability release) Some urgent patches to 3.1.3. Class validation parse bug fixed. Global zone handling error for solaris fixed. Package architectures handled correctly (bug #456). Reading and writing of key name "root-.pub" eliminated (bug #442, #453). cf-serverd crash because of race condition on SERVER_KEYSEEN fixed. Lock purging to avoid remnant complexity explosion (bug #430). Some copyright notices added that got lost. 3.1.3 (Stability release) Major memory leaks in cf-monitord, cf-execd, cf-serverd fixed (bug #427). The daemons now show no growth even with very complex policies. cf-serverd crash due to race condition in DeleteScope() fixed (bug #406). Added 30 second timeout on recv() on Linux. package_noverify_returncode implemented (bug #256). A flexible mechanism for setting classes based on return codes of commands has been introduced. Allows for setting promise kept, repaired or failed based on any return codes. This is currently implemented for commands-promises, package-manager commands and transformer in files. In classes body, see attributes kept_returncodes, repaired_returncodes, failed_returncodes (bug #248, #329). New function ip2host - reverse DNS lookup (bug #146). 3.1.2 (Scalability/efficiency release) Big efficiency improvements by caching output from cf-promises. Can also be used for much more efficient policy deployment (only pull if changed). Caching state of ps command for greater efficiency. Reloaded for each bundle. Index class lookup improves efficiency of class evaluation for huge configurations. Fixed issue where certain promiser strings got corrupted. Minor memory access issues fixed. Iterator bug introduced in 3.1.0 fixed 3.1.1 (Bugfix release) Memory leaks in server tracked down and fixed. List expansion bug (one list items not executed) fixed. Security issue introduced by change of runcommand shell policy fixed. If users defined a runcommand for cf-runagent/cf-serverd communication, possible to execute commands. cf-key -s command for showing key hash/IP address identity pairs 3.1.0 Change in storage of public keys. Cfengine now hashes the public key and uses this as the keyname. Keys will be converted automatically. The old dynamic addresses lists are deprecated. Caching of dns and key information for greater server speed. Change in last-seen format reflects the public key usage. New package policy addupdate - installs package if not there and updates it otherwise. Support for package_changes => "bulk" in file repository as well. New special function readstringarrayidx, similar to readstringarray, but uses integer indices. Very useful if first row elements are not good identifiers (e.g. contains spaces, non-unique, etc.). Change two log formats to use time() instead of date() - filechanges - total compliance Change from using md5 to sha256 as default digest for commercial version, community retains md5 for compat. Commands not returning 0 in commands-promises are flagged as repair_failed. Adjustable timeout on connect(). Defaults to 10 seconds, adjustable with default_timeout in agent control. Redesign of the knowledge map infrastructure. Now possible to use variables to call methods, e.g methods: "name $(list)" usebundle => $(list)("abc"); See reference manual notes Changes to normal ordering to optimize execution. Increased stability by always initializing Attribute and Promise structures. When running cf-promises in dry-run mode (-n), the user does not need to put binaries in WORKDIR/bin. For example, non-privileged users can verify root policies. Source control revision added in version string if run in verbose mode (e.g. "cf-promises -vV"). This needs some refining, uses revision of a header now. New semantics in return values of list functions. Null values are now allowed and there is no iteration over empty lists. The value "cf_null" is reserved for use as a null iterator. 3.0.5p1 Showing paths allowed/denied access to when cf-serverd is run in verbose mode. Bug in server fixed for dynamic addresses. File handle closure bugfix - too many open databases. Seg fault in mount files fix. Twin used in cf-execd without checking. Check_root set wrong directory permissions at source not destination. Error message degraded in body definition. Undefined body not warned as error. Various build enahncements. Package_list_update called only once per manager, and fixed crash. Version number bug in packages. 3.0.5 Encryption problems fixed - client key buffer was uninitialized. Classes-promisers are now automatically canonified when class strings are defined, to simplifying the use of variables in classes. New scalars sys.cf_version and sys.nova_version that hold Cfengine version information. Attribute package_delete_convention added, to allow customizable package name in delete command during update. package_list_update_ifelapsed limit added. Private variable $(firstrepo) is available in package_name_convention and package_delete_convention in order to expand the full path to a package, which is required by some managers. Some of the threading code is rewritten and made more robust. This includes synchronizing access to the lastseen database from the server. Bad initialization of BSD flags fixed Multiple variable expansion issues in control fixed for server and agent Allow ignore_missing_bundles to affect methods: bundles too Run agent trust dialogue fixed Bug in CPU monitoring, increasing time scale caused linear decay of CPU measurement. Bug in Setuid log storage, fix. Hooks added for new Nova virtualization promises. Multithreading mutex failed to collide during cfservd leading to dropped authentication under heavy load. 3.0.4 Class cancellation in promises to create better class feedback, allows emulation of switch/case semantics etc Value of SA measurement promises Special function getenv() which returns the contents of an environment variable (on all platforms). New function translatepath for generic Windows New function escape() to escape literals as regular expressions (like SQL) New function host2ip for caching IP address lookup New function regextract for setting variables with backreferences New variables for the components $(sys.cf_agent), $(sys.cf_know) etc pointing to the binaries. More robust integrated database implementation; closing all handles when receiving signals, self-healing on corruption. Package installation on localhost without a manager like yum completed, multiple repositories searched, and universal methods. Numerous bugfixes 3.0.3 sha256 .. new hashes in openssl included in syntax tree. End of line autocropping in readfile (hopefully intelligent) hashmatch function incorrectly implemented - old debugging code left behind. Fix. sys.crontab variable Unknown user is now interpretated as "same user", so that we give cfengine a chance to fix Unregistered addresses no longer report "(Non registered IP)", but return as the address itself when doing reverse lookups. 3.0.2 IMPORTANT: Change in normal ordering of editing. replace comes after insert lines Much testing and minor bug fixing Memory leaks fixed Many hooks added for Nova enterprise extensions. promise_output reports now placed in WORKDIR/reports directory Initialization correction and self-correx in monitord Many new body constraints added. Code readied for enterprise version Nova. -b option can override the bundlesequence (must not contain parameters yet) collapse_destination_dir option added to copy so that files can be aggregated from subdirectories into a single destination. Preparation for release: unit_accessed_before.cf x unit_accumulated_time.cf x unit_acl.cf x unit_acl_generic.cf x unit_ago.cf x unit_arrays.cf x unit_backreferences_files.cf x unit_badpromise.cf x unit_badtype.cf x unit_bsdflags.cf x unit_cf2_integration.cf x unit_changedbefore.cf x unit_change_detect.cf x unit_chdir.cf x unit_classes_global.cf x unit_classmatch.cf x unit_classvar_convergence.cf x unit_compare.cf x unit_controlclasses.cf x unit_control_expand.cf x unit_copy.cf x unit_copy_edit.cf x unit_copylinks.cf x unit_createdb.cf x unit_create_filedir.cf x unit_definitions.cf x unit_deletelines.cf x unit_disable_and_rotate_files.cf x unit_dollar.cf x unit_edit_column_files.cf x unit_edit_comment_lines.cf x unit_edit_deletenotmatch.cf x unit_edit_insert_lines.cf x unit_edit_insert_lines_silly.cf x unit_edit_replace_string.cf x unit_edit_sectioned_file.cf x unit_edit_setvar.cf x unit_edit_triggerclass.cf x unit-env.cf x unit_epimenides.cf x unit_exec_args.cf x unit_execd.cf x unit_exec_in_sequence.cf x unit_execresult.cf x unit_expand.cf x unit_failsafe.cf x unit_file_change_detection.cf x unit_fileexists.cf x unit_file_owner_list_template.cf x unit_fileperms.cf x unit_filesexist2.cf x unit_filesexist.cf x unit_getgid.cf x unit_getindices.cf x unit_getregistry.cf x unit_getuid.cf x unit_global_list_expansion_2.cf x unit_global_list_expansion.cf x unit_groupexists.cf x unit_hash.cf x unit_hashcomment.cf x unit_hashmatch.cf x unit_helloworld.cf x unit_hostrange.cf x unit_intarray.cf x unit_iprange.cf x unit_irange.cf x unit_isdir.cf x unit_islink.cf x unit_isnewerthan.cf x unit_isplain.cf x unit_isvariable.cf x unit_iteration.cf x unit_knowledge_txt.cf x unit_lastnode.cf x unit_ldap.cf x unit_linking.cf x unit_literal_server.cf x unit_locate_files_and_compress.cf x unit_log_private.cf x unit_loops.cf x unit_measurements.cf x unit_method.cf x unit_method_validate.cf x unit_module_exec_2.cf unit_module_exec.cf unit_mount_fs.cf x unit_neighbourhood_watch.cf x unit_null_config.cf x unit_occurrences.cf x unit_ordering.cf x unit_package_apt.cf x unit_package_hash.cf x unit_package_rpm.cf x unit_package_yum.cf x unit_package_zypper.cf x unit_parallel_exec.cf x unit_pathtype.cf x unit_pattern_and_edit.cf x unit_peers.cf x unit_postfix.cf x unit_process_kill.cf x unit_process_matching2.cf x unit_process_matching.cf x unit_process_signalling.cf x unit_readlist.cf x unit_readtcp.cf x unit_regarray.cf x unit_registry.cf x unit_regline.cf x unit_reglist.cf x unit_remove_deadlinks.cf x unit_rename.cf x unit_report_state.cf x unit_reporttofile.cf x unit_returnszero.cf x unit_select_mode.cf x unit_select_region.cf x unit_selectservers.cf x unit_select_size.cf x unit_server_copy_localhost.cf x unit_server_copy_remote.cf x unit_server_copy_purge.cf x unit_splitstring.cf x unit_sql.cf x unit_storage.cf x unit_strcmp.cf x unit_stringarray.cf x unit_syslog.cf x unit_template.cf x unit_tidy_all_files.cf x unit_user_edit.cf x unit_user_edit_method.cf x unit_userexists.cf x unit_varclass.cf x unit_vars.cf x unit_warnifline.cf x unit_webserver.cf x 3.0.1 First standalone release, independent of cfengine 2 Purge old definitions and check consistency. NB: changed search_mode to be a list of matching values Reporting rationalized in cf-promises with -r only to avoid leaving output files everywhere. Hooks added for upcoming commercial additions to cfengine. Added classify() and hostinnetgroup() functions Added additional change management options for change detection Package management added - generic mechanisms. Limits on backgrounding added to avoid resource contention during cfengine runs. Image type added to cf-know. New classes for quartly shifts: Morning,Afternoon,Evening,Night Bug fixes in editfiles - line insertion for multiple line objects Change the name of the variables and context from the monitord for better separation of data, and shorter names. sys -> mon average -> av, stddev -> dev canonical name for windows changed from "nt" to "windows", also version names added "vista","xp" etc.. License notices updated for dual license editions. 3.0.0 First release of cfengine 3. Known omissions: - no support for ACLs - no support for packages - no support for interface configuration These will be added in the next release.
Enterprise ChangeLog
For the complete history of Enterprise-specific changes in the CFEngine version you have
installed, see the ChangeLog.Enterprise
file in /var/cfengine/share/doc
.
3.6.7 Fixes: - Removed error message from cf-serverd when not finding software inventory. E.g. "Failed to access current state for report: 'software'". - Introduce by field truncation for promise execution entries. (Redmine #7466) - Fix error logging to MP via HTTPS. (Redmine #7687) - Fix scheduled report not beeing emailed when report type is set to only contain CSV file type (Redmine #3780, #7619) - Fix resource restrictions of SQL API matching table names as substrings. (Redmine #7536) - Forbid access to status table using SQL API 3.6.6: Bug fixes: - Ignore empty log messages while logging promise executions in cf-agent evaluation. - Fix Postgres CPU usage spikes. - Fix upgrate for monitoring. - Fix duplicate key value violates unique constraint "status_pkey" error. - Reduce database size in high load hub by making vacuum strategy more aggressive. - Process matching on Windows has been rewritten, which should make process promises work more reliable there. (Redmine #6977) - Failure in output log cleanup on Windows has been fixed. (Redmine #7149) - CFEngine no longer waits an entire "call_collect_interval" before doing the first report collection. (Redmine #5436) - Add truncation for promise attribute sizes to prevent from ignoring to long reports. 3.6.5: Bug fixes: - Fix cleaning-up monitoring during upgrade. - Remove unused bundles.lmdb to reduce agent I/O usage. - Redesign classes and variables storage (for reporting) to reduce I/O usage. - Improve API performance for DELETE requests on /api/host/:id resources. Mission Portal: - Small CSS changes - Widgets & alerts view - UI changes - Updated links to support portal 3.6.4: Bug fixes: - Fix "cfe_autorun_inventory_dmidecode" error message on Windows if Powershell is not installed. - Fix bogus failed promise, "cfe_internal...", as a result of indexing packages for the inventory screen. (Redmine #6865) Mission Portal: - LDAP settings UI improvements - Unsaved SQL and Inventory Reports are preserved while refreshing/navigating in browser browser - Help text: Added instructions to turn on Monitoring data - Health bar dropdown labels and reports renamed Changes: - Introduce automatic rebase for the client if the client have not been successfully collected for defined period of time. Timeout is set by client_history_timeout attibute in hub body and if it is not set, it defaults to 6 hours. Note: During rebase all accumulated reports up till that event are ignored and not collected by the cf-hub. 3.6.3: Mission Portal: - Added license information to header - General UI cleanups and small bug fixes - Optimization of Software Updates alert - Inventory reports: - Made software filtering case insensitive - Updated help text - Performance improvements - Alerts: - Bug fix for duplicate alerts in overview - Bug fixes for deleting alerts & widgets - Settings: - LDAP search filter help text & validation 3.6.2: Changes: - Monitoring magnified and monitoring yearly database schema have been redesigned to reduce database disk space usage over time. - RBAC backend have been redesigned from dynamically generated tmp views to static global views that use session variables for passing context filters and host identifier. Filtering also switched from dynamically generated queries to Full Text Search. Mission Portal: - UI changes: redesigned alerts + conditions overview screen - Layout improvement of alert results view - Added navigation menu buttons to dashboard + alerts screens - High Availability status added to header bar - Custom notification script UI added to settings and alert editing - Added 'Low disk space' alert + 'System health' widget OOTB - Bug fixing/small UI improvements 3.6.1: Changes: - 'cf-key --install-license' installs hub-specific license key file "fqname-hostkey.dat" in $WORKDIR/licenses, where they can easily be managed centrally via a VCS - hub-specific license file is searched in $WORKDIR/licenses before license.dat is searched in $WORKDIR, $WORKDIR/inputs and $WORKDIR/masterfiles - Where appropriate, Enterprise API returns proper NULL json objects rather than literal "NULL" values Mission Portal: - streamlined UI for inventory reporting - fix username/role lookup failures if external authentication backend is case insensitive - reduce number of LDAP roundtrips - allow filtering of reports by category - allow reordering of widgets on dashboard - UI for bulk-deleting decommissioned hosts from "health" menu - various behind-the-scene fixes and improvements to speed up UI and reporting Platform support: - Introduced Windows support into the CFEngine 3.6 series. 3.6.0 Mission Portal: - UI and layout improvements and cleanups - Dashboard and alerts introduced - Inventory report type and view introduced - Report categories introduced - CFEngine health indicator added to UI with links to associated reports - Host number indicator added to UI - Inline help and help pop-ups added for new features - Welcome tour pop-up introduced - Host filter UI improvements - search host name, select/deselect all - About CFEngine page - license and version information has moved to a dedicated page in settings - Fixes for IE8 compatibility - Added option in UI to allow logging in to Mission Portal over https - Design Center sketch catalog redesign - sketches can now be filtered by category, tag, or search - UI to reset git settings in Design Center Hub: - Remove --cache / -a command line option from cf-hub binary - Remove --index / -i command line option from cf-hub binary - Remove --maintain / -m command line option from cf-hub binary (Maintenance process have been implemented in the policy) - Remove MongoDB Diagnostics - Promise repaired/notkept log have been removed from report collection. It have been replaced by promise executions report. - Total compliance report have been removed from report collection. - Setuid report have been removed. - Promise definitions report have been removed. - Promise and bundle compliance reports have been removed. Their functionality have been replaced with promise executions report. - Reporting database and report collection architecture have been redesigned to improve performance and scalability characteristics. - MongoDB reporting database have been replaced with PostgreSQL 9.3 - Context, Variable, Software Installed, Software Patches and Promise Execution reports support history over time. History length is controlled per report type and can be configured in cfe_internal_hub_maintain bundle. - Introduce new hub query type: 'rebase' ('full' query aliases 'rebase') for re-downloading full state of the client in current moment. Rebase query result overwrites all non-historical entries about the host in the database. - Include meta data contents to contexts and variables reports. Enterprise Rest API: - Rest interface for Design center - Additional information returned for host (lastreport and firstseen) - Rest API 2.2 (/rest API) have been removed. - Enterprise API performance have been improved. - SQL API table schema have been redesigned. - Remove API cache. - Pagination and sorting improvements. - Introduce 'hostIdentifier' setting to /api/settings. - Fixes in LDAP support. - Delete host API now additionally removes host from lastseen database as also removes host public key. Bug fixes: - Removed MongoDB - cf-serverd for Windows now binds to both IPv4 and IPv6 by default, not just IPv6. (Redmine #3980) - cf-agent now reports host packages installed and available by default. (Redmine #3257) - Fixed incorrect file diff generation when a line had moved within a file, and certain other corner cases. (Redmine #5015) - Windows fixes: - CFEngine now handles Windows newlines correctly within text files when editing or using the module protocol. Existing text files will keep their newline type (either LF or CRLF), whereas new files will get CRLF newlines. (Redmine #4733) - CFEngine will no longer display a blocking popup if it crashes. - CFEngine now reports uptime correctly on Windows. Enterprise extensions: - Remove promise_notkept_log_include, promise_notkept_log_exclude, promise_repaired_log_include, promise_repaired_log_exclude (syntax is valid but not functional) - Remove classes_include, classes_exclude, variables_include, variables_exclude (syntax is valid but not functional) - Introduce promise_handle_include, promise_handle_exclude attributes from report_data_select - Introduce metatags_include, metatags_exclude attributes from report_data_select - Deprecate export_zenoss attribute - Introduce promise_execution.log containing outcome and information about all executed promises. It can be found under cfengine/state/ data format is CSV. - Agent execution time have been included into benchmarks report. - After disabling report_data_select filtering rule, include last known value in next packaged report. 3.5.3 Bug fixes: - purge old data for promises with long promise handles (Redmine #3438) - fix constraint violation in PromiseDefinitions table which resulted in error everytime this table was loaded (Redmine #3370) - enable update of promise definitions database from policy - fix cfengine3 init.d script to correctly detect debian systems with yum installed (Redmine #3589) Mission Portal: - various layout and UI fixes - fix editing of event trackers - speed up listing of hosts for promises not kept - maintain host context (Redmine #3474) - ability to manually add context filter in the SQL app (Redmine #3466) - host identifier settings simplified (Redmine #3101) Packaging fixes: - Correct php.ini path in the packaged httpd (Redmine #3445) - Add missing mongodb tools in ubuntu/debian hub packages (Redmine #3444) - Fix manpath error for SLES (Redmine #3539) - Fix file permissions - some policy files had executable bit set (Redmine #3521) 3.5.2 Changes: - MongoDB has been upgraded to version 2.2.4 - monitoring data has moved into a separate database See db-move-monitoring-to-cfmonitor.js script to migrate data Bug Fixes: - Fix segfault of cf-serverd on HP-UX - Do not to start a mongodb repair unnecessarily - cf-hub -H now supports multiple hosts Mission Portal: - Reports can be published and shared between users - Various UI improvements - Optimizations in the report engine 3.5.1 Mission Portal: - SQL queries can be shared between users - Fix timing issues for downloading large SQL reports - Purge sketch data when no longer used by active sketches - Uninstall sketches that have no activation - Support for boolean, menu option and optional parameters in Design Center UI - UI fixes to user and role management pages - Delete navigation tree definitions of deleted uses - Fixes to password reset - General UI improvments - Fixes for IE8 compatibility Changes: - Perform a database repair from init script if unclean shutdown of mongod is detected - Redmine #3035 - Data collection and cf-hub - Improved database connection handling during report collection by cf-hub. - REST APIs support an optional disableCache flag; when set, the backend always hits the MongoDB - Redmine #2945 Bug Fixes: - don't generate ERR message during maintenance if environments couldn't be queried, changed to INFO message - Fix usemodule on Windows (Redmine #1884) - Fixed replica set detection (regression in 3.5) - Redmine #2806 - Set correct precision format when storing db diagnostics to avoid null-values - Fix possible division-by-zero bug in compliance meters (Redmine #2734) 3.5.0 New features: - Mission Portal - added Design Center UI to simplify sketch activation, including MP specific git settings to support version control of sketch configurations - re-focused apps support quick navigation - added persisting host and policy context between apps - extended the SQL builder interface with more tables - Added FirstReportTimeStamp into Hosts table in SQL REST API. This time value represent fist report time after bootstrap, already bootstrapped agents will set this with first report after update. - Added regular expression support to SQL queries - HostContext filter support in SQL REST API. - added global navigation trees which are only editable by admins, including the option to share trees with other users - REST API extensions - New optional parameters for REST API were added: hostContextInclude and hostContextExclude (array type) - PromiseContext filter support in SQL REST API. New optional parameter for REST API was added: promiseContext (input: all / user / system) - Data collection and cf-hub - Added set and clear triggers for persistently disabling CFEngine components. eg. to disable cf-monitord, run cf-agent with "-Dset_persistent_disable_cf_monitord" to re-enable use: "-Dclear_persistent_disable_cf_monitord" - Host side report content filter for class, variable, promise log and monitoring reports. Controlled by report_data_select body in access promise. - Diagnostics logging and SQL REST API for MongoDB, report collection and maintenance process on the enterprise hub. - Windows - Windows Powershell support. execresult(), returnszero() and commands promises now supports "powershell" as an option in addition to the "noshell" and "useshell" variants. "powershell" is also added as a hard class in order to test whether Powershell is available. Changes: - Mission Portal - new visual design - streamlined interactions for building new trees - trees are now loaded lazily - general clean-up to the tree controls - hosts in trees are no longer color coded - hosts are only classified as red, green or missing data - operating system tree is now loaded by default - SQL queries are now run by default after clicking their respective links, running a query is now primary action in the UI (#2393) - data and result sets can now by filtered based on navi-tree - logged-in user's name is visible again in the toolbar - removed beta apps - Windows - Improved ACL handling on Windows, which led to some syntax changes. We now consistently use the term "default" to describe ACLs that can be inherited by child objects. These keywords have received new names: acl_directory_inherit -> acl_default specify_inherit_aces -> specify_default_aces The old keywords are deprecated, but still valid. In addition, a new keyword "acl_inherit" controls inheritance behavior on Windows. This feature does not exist on Unix platforms. - Enterprise API - Export SQL results to sqlite3 database file - Data Collection and cf-hub - cf-hub has got an option -q to query reports from the running agent. This option used to reside in cf-runagent, and has been moved to cf-hub. - Full and delta reports send only mon and sys variables as also hosts excluding policy server are reporting only subset of monitoring data. This can be changed using access promise in default cf_serverd.cf policy. - Removed license checking on hosts. Bugfixes: - Mission Portal - promise finder now does string matching - assigning roles in user management now makes sense - report builder now has a "new query" button - blue hosts' lacking data history is indicated correctly now - CSS fixes - finders no longer load duplicate list items - Fix inconsistent behavior of black host status directly after install. - Enterprise API - Fix some REST queries not working on replica secondaries (eg. the /rest/host/:id) - Remove HTML output from Total Compliance report 3.0.x Removed unused options "[-t][-r][-u]" from cf-know 3.0.1 $(sys.licenses_installtime) variable removed from "Enterprise Free" 3.0.0 New Reporting Engine: A SQL interface to reports collected by hub. We allow all standardised SQL SELECT constructs to query the SQL reports database, with the following additions: - TIMESTAMP_UNIX() - seconds elapsed since 1970 - TIMESTAMP_UNIX_DAYS() - days elapsed since 1970 These are added to avoid use of non-portable SQL date/time functions. Enterprise API: Read + write REST interface for - report querying(utilizes underlying reporting engine) - user management - can be used with REST API v1 in parallel Configurable hostnames(host/system identifier) in reports - can take any of the sys variables(eg.$(sys.fqhost)) When decommissioning (deleting a host) from the Mission Portal/Enterprise API, the public keys of the clients are also removed Ability to delete multiple hosts from the mission portal Improvements on the hub maintenance process - less resource intensive and configurable - New option for cf-hub added (-m) for Enterprise database maintenance Fixes on database connections problems - If you were seeing "connection refused because too many open connections", in database log please consider upgrading Changed "nova>" to "enterprise>" in agent verbose output - Please update email filters Removed internal CFE promises from reporting New classes enterprise, enterprise_X, enterprise_X_Y, enterprise_X_Y_Z on CFEngine Enterprise, to reflect the version running. New variable sys.enterprise_version that holds the CFEngine Enterprise version. This complements the Nova classes and the sys.nova_version variable, which will eventually be deprecated. Fix file change report containing warning message as filename for new/deleted files File diff log (nova_diff.log) have been extended with promise handle name. Total compliance output in cf-agent verbose mode and promise_summary.log have been extended with user and cfengine internal compliance level. System variables are collected by hub in every delta query Fix software reports showing "(never)" in the "Last seen" column Fix "blue hosts" list being empty for clients that don't have class keys Sendmail is installed by default on the hub - required for emailing of reports 32-bit hub installations no longer supported Created a variable update_policy.mongodb_dir, for cases where MongoDB should not run out of /var/cfengine/state (could grow to tens of gigabytes). Removed commercial_customer class, as it was unused in internal policies. Please use enterprise_edition instead if you used this in your policies. New performance report events: DBPurgeHostsAll, DBMaintenance, DBMaintenanceTimestampsSingleHost, DBCacheCompliance, DBReportCollectAll. License verification is made more robust by not relying on the last-seen database anymore. This means you do not need to bootstrap a client to verify the license. See the cf-key --install-license option. 2.2.0 More diagnostics on report collection from cf-hub. Logging more useful information in cf-hub -l, measuring total collection time in benchmarks report, under id "ReportCollectAll". Fixed issue where client would show as green in the Mission Portal when no data was received, e.g. due to access or license error at client. Now correctly shows as blue in these cases. Greatly reduced amount of connections from cf-hub to localhost mongodb. Now there is one connection per cf-hub run (max 50), before it was three per client. Software and variable report now contains end-node discovery time. Software and patches available/status reports contain maximum 5-minute old data, improved from 6 hours in last release. This will only apply to clients that are upgraded to 2.2.0. Software report query from Mission Portal is much faster on larger data-sets due to removal of autocomplete feature. Upgraded mongod from version 1.8.2 to 2.0.4, which increases efficiency on concurrency and reduces memory usage. New function hostswithclass() that generates a list of hosts in a given class on the hub. The bundled failsafe.cf policy now has trustkey=false to avoid IP spoofing attacks in default policy 2.1.0 New interface with interactive graphs on Nova hub. User management on hub. Allowing TAB in file diff report. Better handling of large diffs and attempts to diff binaries. Support for adding notes to all hosts and reports. Support for policy staging environments. Nova reports can be exported to file and imported manually using cf-report -x and cf-report -i. Faster collection of monitoring (vitals) data due to new protocol and data structure. Long-term storage (one year) of diff and changes reports. The variables report got a last-seen column. Now variables are stored in the hub for a longer time like classes, and not overwritten on every update. The report of reports-promises are shown in the promise repaired log in the Mission Portal. New option cf-hub --cache, recreates the cache data needed by the web interface. Only showing the last seen host name and ip address when listing hosts and on the host page. Reliability improvements, especially when querying the variables report. 2.0.2 Much faster report querying. Allows to specify age interval when querying promise not kept and repaired. Promiser conflict identifcation. Built with Cfengine Community Edition 3.1.4. 2.0.1 Fixed promise query of not kept/promise repaired logs. 1.2.0 Windows Event Logs include output_prefix if set, and which component reported the event. The verbosity of event logs have been reduced by not including promise kept and repaired events, this can now be tuned with action.log_level. Take out network communications from total state calculation, as it gets counted twice. 1.1.2 Encryption problems fixed in Community Edition. Built with Cfengine Community Edition 3.0.4p3. 1.1.0 Regular expressions in file paths supported on Windows by using forward slash as path separator. CPU utilization report on Windows. Users logged in report on Windows. On the Windows cf-serverd, requests for /var/cfengine are translated to $(sys.workdir)\Cfengine, and path separators are automatically adjusted ("/" becomes "\"). This yields support for more platform-independent promises and allows for automatic copying of reports from Windows clients to the policy server. Scale on graphs in the Knowledge Map, and different background color gives more readability. Special functions added for accessing remote classes for distributed cooperation. 1.0.0 Reports added to cf-report for compliance, setuid, file_changes etc. Added csv format also Automating topic map integration of policy, with impact analysis using promisee and builds_on promises. Literal string lookup in server. Database SQL and registry functions added. Verification and sanity checking of SQL database table structure. Create and destroy databases convergently. Access control list support for Linux. Powerful and lightweight promises for Customizable monitoring and system discovery promises added to cf-monitord. Longterm memory for 3 year trend analysis.
Supported Platforms and Versions
CFEngine works on a wide range of platforms, and the CFEngine team strives to provide support for the platforms most frequently used by our users.
Enterprise Server
Platform | Versions | Architecture |
---|---|---|
CentOS | 5, 6 | x86-64 |
Debian | 6, 7 | x86-64 |
RHEL | 5, 6 | x86-64 |
SLES | 11 | x86-64 |
Ubuntu | 10.04, 12.04 | x86-64 |
Hosts
Any host can be a policy server in Community installations of CFEngine.
OS | Versions | Architectures |
---|---|---|
AIX | 5.3*, 6, 7 | PowerPC |
CentOS | 4, 5, 6, 7** | x86-64, x86 |
Debian | 6, 7 | x86-64, x86 |
HP-UX | 11.23+ | Itanium |
RHEL | 4, 5, 6, 7** | x86-64, x86 |
SLES | 10, 11 | x86-64, x86 |
Solaris | 9 | SPARC |
Solaris | 10, 11 | UltraSparc |
Ubuntu | 10.04, 12.04 | x86-64, x86 |
Windows | 2008 | x86-64, x86 |
Windows | 2008, 2012 | x86-64 |
* AIX 5.3 is required to have "5300-05-CSP" or later
** RHEL 7 and CentOS 7 are currently not supported as hubs, see Known Issues
Known Issues also includes platform-specific notes.
CFEngine Enterprise has Virtual I/O Server (VIOS) Recognized status from IBM. This means that CFEngine Enterprise has been technically verified by IBM to be installed in and manage VIOS environments.
Hub/Host compatibility
An upgrade path from previous versions is available from 3.6.1.
Some data will not be available from older hosts, and the policy you serve needs to take into account hosts with different versions.
Future platform support
The CFEngine team will continue to support future releases of popular Host platforms, including RHEL, Debian, Ubuntu, as well as maintaining support for existing platforms important to users.
In general, CFEngine is known to run on a wide range of other platforms. As long as the platform is POSIX compliant and has a C compiler toolchain that fully implements the C99 standard, we are happy to work with you to make CFEngine available. Please contact our sales team for details.
Policy Framework Updates
CFEngine Policy Framework Updates for 3.6
If you follow the CFEngine masterfiles policy framework (the masterfiles you get out of the box) we encourage you to upgrade the policy framework each time you upgrade CFEngine. We recommend making as few changes as possible to the shipped masterfiles to make these upgrades as painless as possible. Generally the best way to accomplish that is to take your custom policy and integrate it on top of the new masterfiles.
3.6 introduces significant changes to the masterfiles policy framework. Masterfiles was moved out from core into its own repository and was merged with the policy framework from CFEngine Enterprise. Now both the core (CFEngine Community) and CFEngine Enterprise ship with a common masterfiles policy framework. For us it makes maintanence easier, and for the community and customers it ensures that improvements that come into the framework are easily accessible from either product.
Please consult The Policy Framework for a map to the policy framework.
What is new in the 3.6 masterfiles policy framework
Directories
templates
Many users create a top level templates directory for global
template distribution. We made it easy by creating a templates/
directory
that's always copied for you.
cfe_internal
The cfe_internal/
directory contains policies that manage how CFEngine
runs, much of this is in relation to the Enterprise version.
inventory
The inventory/
directory contains policies that do discovery of various
inventory attributes.
sketches
The sketches/
directory is where Design Center sketches and their
associated runfile get deployed.
update
The update/
directory contains policies that have been split out from
the previously monolithic update.cf
.
Features defined in def.cf
def.cf
has continued various global settings
for some time, including various common directories, and simple access control
list definition to control which remote hosts could connect. Comparing def.cf
from 3.5 and 3.6 will reveal many additions
- services_autorun provides automatic loading and activation of policy.
- cfengine_internal_rotate_logs Enables log rotation for CFEngines log files.
cfengine_internal_encrypt_transfers Enables encryption for policy and binary updates done during the update policy.
Note: This setting is mirrored from update.cf for CFEngine enterprise reporting. This setting is superfluous when
protocol_version
is set to 2 or higher which enables TLS encryption for communication.cfengine_internal_purge_policies: Enables purging of policies that no longer exist in masterfiles.
Note: This setting is mirrored from update.cf for CFEngine Enterprise reporting.
postgresql_maintenance: Enables database maintenance for CFEngine Enterprise Policy hubs.
def.cf
also sees the addition of bundle common inventory_control
which is
responsible for controlling the new inventory bundles. Inventory bundles
provide variables and classes that can be useful for both Enterprise reporting
and directly from within policy. LSB (Linux Standards Base), Dmidecode, LLDP
(Link Layer Discovery Protocol), software installed, mounted filesystems
(mtab), configured filesystems (fstab), and proc are some of the inventory
policies provided out of the box.
See The Policy Framework to understand inventory modules and learn how you can use the inventory information from within other policy and from the CFEngine Enterprise Mission Portal.
The Standard Library
The standard library lib/3.6/*.cf
has seen many improvements. First and foremost is the addition of doxygen
style inline documentation. This inline documentation markup is levereged by
our documentation system to provide a rendered version of the standard library
which is included in the reference manual.
You only need to include $(sys.libdir)/stdlib.cf
to get the whole
standard library. sys.libdir
resolves to the right path for your
installation, including the version (3.6, 3.7, etc.)
Of special note: the
standard_services
has been improved to make it more dynamic, and now has
support for systemd, sysvinitd, sysvservice, and chkconfig. New convenianence
bundles and bodies for common patterns have been added including:
dir_sync()
file_copy()
file_empty()
file_hardlink()
file_link()
file_make()
file_mustache_jsonstring()
file_mustache()
file_tidy()
converge()
fstab_option_editor()
insert_ini_section()
prepend_if_no_line()
resolvconf_o()
set_line_based()
fstab_options()
bigger_than()
linkfrom()
package_absent()
package_latest()
package_present()
package_specific_absent()
package_specific_latest()
package_specific()
package_specific_present()
brew()
npm()
pip()
process_kill()
by_owner()
logrotate()
prunedir()
cmerge()
rm_rf_depth()
url_ping()
setuidgid_dir()
setuid_gid_umask()
system_owned()
update.cf
The update.cf
policy got broken up from its monolithic form and
now has additional update policies included inside the update
directory.
bundle common update_def was added to provide a common place to enable
features related to policy updates (much like body common def found in
def.cf
). Please see The Policy Framework for the full details.
- cfengine_internal_masterfiles_update: This class enables automatic policy deployment on the policy server. This is designed for use with CFEngine Enterprise.
- cfengine_internal_encrypt_transfers Enables encryption for policy and
binary updates done during the update policy. Note: This setting is mirrored
in def.cf for CFEngine enterprise reporting. This setting is superfluous when
protocol_version
is set to 2 or higher which enables TLS encryption for communication. - cfengine_internal_purge_policies: Enables purging of policies that no longer exist in masterfiles. Note: This setting is mirrored in promises.cf for CFEngine enterprise reporting.
- ...and much more!
Now that you have an overview of whats new, lets cover a few things you need to watch our for when upgrading.
Upgrade Gotchas
General
Array Keys are not iterated in the decalred order any more
The order of results returned by getindices function is not necessarily returned in the order defined. Example:
bundle agent example
{
vars:
"array[key1]" string => "value1";
"array[key2]" string => "value2";
"array[key3]" string => "value3";
"keys" slist => getindices(array);
reports:
"$(keys)";
}
The above policy will produce the following output on 3.5:
R: key1
R: key2
R: key3
The same policy in 3.6 will produce:
R: key1
R: key3
R: key2
If the resulting list order is important please consider the sort()
and
reverse()
functions.
Function Caching
3.6 introduces function caching(https://docs.cfengine.com/docs/master/reference-functions.html#function-caching) to help further improve performance and execution speed.
This behavior can be disabled by setting the cache_system_functions
common
control attribute to "false". A function is cached when it is first evaluated
and it's cached result is used until the end of the agent execution. If have
previously depended on function re-evaluation in order to have proper results
after an action has been taken consider guarding the function call to only be
evaluated after a pre-conditional promise using conventional class guards,
or the ifvarclass
and depends_on
attributes.
Files considered by update.cf have moved and changed
Previously body file_select u_input_files
controlled which files were
considered when performing a policy update. That list of extensions has moved
to update_def.input_name_patterns
. '*.conf', and '*.mustache' files have
been added to the default list.
For more information, see the [FAQ][FAQ#i-have-added-new-files-in-masterfiles-but-my-remote-clients-are-not-getting-updates]
Enterprise Users
Options controlling which variables to report have moved and changed
In 3.6 we begin using tags to identify which variables and classes to report
on. report_data_select
has moved from controls/cf_serverd.cf
to
lib/$(sys.cf_version_major).$(sys.cf_version_minor)/reports.cf
. In 3.6 we
report on variables and namespace
scope classes that are tagged with
"inventory" or "report" using the new metatags_include
and
metatags_exclude
attributes.
The following attributes have been deprecated in 3.6. Please tag the variables and classes you would like to report on appropriately. If you have any questions please contact support.
classes_include
classes_exclude
variables_include
variables_exclude
promise_notkept_log_include
promise_notkept_log_exclude
promise_repaired_log_include
promise_repaired_log_exclude
Host licenses paid deprecated
This option was deprecated in CFEngine 3.5 and is no longer valid syntax.
This information is embedded in your Enterprise license, simply ensure that
host_license_paid
is not defined in your body common control
.
Known Issues
CFEngine defects are managed in our bug tracker. Please report bugs or unexpected behavior there, following the documented guideline for new bug reports.
The items below highlight issues that require additional awareness when starting with CFEngine or when upgrading from a previous version.
Windows agents are unable to use the shortcut feature
Windows agents are unable to use the shortcut feature in access type promises. This is resolved in 3.7.x and will not be backported to 3.6.x.
Constant variables do not work, variables are always overrideable
https://dev.cfengine.com/issues/1492
Showing Classes and variables with cf-promsies
cf-promises --show-classes
and cf-promises --show-vars
will only show
classes and variables found on a first pass through the policy, since
cf-promises
does not evaluate agent promises.
Protocol incompatibility between
The CFEngine protocol versions 1 and 2 are incompatible (the latter is based on TLS). CFEngine 3.6 supports both protocol versions, but earlier versions only support protocol version 1. Protocol version 1 is still the default in 3.6 but the default will change to 2 in future versions.
RHEL 7 / CentOS 7
- The CFEngine Enterprise hub does not yet support RHEL 7 / CentOS 7
- The community package repositories hosted by CFEngine does not have the RHEL 7 /CentOS 7 package. Please download and install the package separately.
- If you use the community edition as a Policy Server on RHEL 7 / CentOS 7 you will see "Failed to start the server" when you bootstrap it to itself. This message is harmless and will be removed in future releases.
HP-UX specific
- Package promises do not have out-of-the-box support for the HP-UX specific package manager. The workaround is to call the package manager directly using commands promises.
- Some important system information is missing from the HP-UX inventory report, as well as from CFEngine hard classes and system variables. The workaround is to use system tools to obtain the required information and set classes based on this. * Disk free * Memory size * Several OS and architecture specific attributes * System version * System serial number * System manufacturer * CPU model * BIOS version * BIOS vendor
- User promises do not work reliably on HP-UX. It is recommended not to use them at this time. The workaround is to call the useradd/usermod/userdel tools directly. (fixed in CFEngine 3.6.4)
- Process promises depend on the
ps
native tool, which by default truncates lines at 128 columns on HP-UX. It is recommended to edit the file/etc/default/ps
and increase theDEFAULT_CMD_LINE_WIDTH
setting to 1024 to guarantee that process promises will work smoothly on the platform. edit_xml
bundles do not work on HP-UX. (fixed in CFEngine 3.6.4)- Upgrading CFEngine on HP-UX is not supported by the out-of-the-box policy. There is a support article with a workaround.
Enterprise Mission Portal is slow and/or /var/cfengine/state/pg consumes a lot of space/iops or CPU utilization is high
With ceratin policies, the BenchmarksLog table in the PostgreSQL database is known to grow large with CFEngine versions from 3.6.0 to and including 3.6.5. This issue is resolved in CFEngine 3.6.6 and later versions. To test for the problem, run the following commands on the hub:
# /var/cfengine/bin/psql cfdb
SELECT nspname || '.' || relname AS "relation", pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size" FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN ('pg_catalog', 'information_schema') AND C.relkind <> 'i' ORDER BY pg_total_relation_size(C.oid) DESC LIMIT 20;
If the table public.__benchmarkslog is larger than 2-3 GB, you are likely affected by this isse. The table is not used unless you have created custom queries against it yourself. To resolve the issue, please run these commands after a fresh install of any hub prior to version 3.6.6:
# /var/cfengine/bin/psql cfdb
cfdb=# truncate table __benchmarkslog;
cfdb=# create index benchmarks_subselect ON __benchmarkslog (hostkey, eventname, checktimestamp) WITH (FILLFACTOR = 70);
If your hub is still slow or BenchmarksLog was not the problem, please contact support.
Unsupported direct upgrade from CFEngine Enterprise 3.6.0 / 3.6.1 to 3.6.5
Due to an issue in Enterprise 3.6.5 direct upgrade from 3.6.0 and 3.6.1 may resolve with database schema curruption. Upgrade from 3.6.0 / 3.6.1 to 3.6.5 can be done by upgrading first to 3.6.2 / 3.6.3 / 3.6.4 and then to 3.6.5.
Enterprise emails sent for alert noticies come from 'admin@orginization.com'.
There is currently no setting in Mission Portal to configure the sender email address. This issue is tracked in ENT-695 and will be addressed in a future release.
To change the setting you must edit the from email address in
/var/cfengine/share/GUI/application/config/appsettings.php
. Policy in the
Masterfiles Policy Framework will take care of updating the running config
during the next policy run.
// Default FROM email address
$config['appemail'] = 'admin@organisation.com';
Enterprise monitoring graphs
Monitoring graphs are disabled by default in CFEngine Enterprise 3.6 and later versions. To enable them, change monitoring_include in masterfiles/lib/VERSION/reports.cf to e.g. ".*". Note that this has a significant impact on the resource consumption of your hub.
Monitoring graphs are not supported on all platforms, currently Aix and Windows do not have this data.
Scheduled reports not written to expected location
There is a known issue when generating scheduled reports for ONLY CSVs. The
report gets written to /var/cfengine/reports
instead of
/var/cfengine/httpd/htdocs/tmp
and it is NOT named for the report title,
but
instead named by internal identifiers (like
admin-17_admin_1440692523-1441038374.csv).
The workaround is to schedule BOTH CSV and PDF reports. When both CSV and PDF
reports are scheduled the CSV report will be written to
/var/cfengine/httpd/htdocs/tmp
and named for the report title as expected.
Enterprise reports not collected from 3.5
CFEngine Enterprise 3.6 has a new diff-based report collection mechanism, and so a 3.6 hub cannot collect reports from 3.5 or earlier agents.
Currently the 3.5 agents will not show in Mission Portal at all, but
you will see them by running cf-key -s
on the hub.
Enterprise software inventory is not out-of-the-box
Software inventory is not out-of-the-box for reporting from the hub on Windows platforms. In order to add it, please contact support for a custom policy.
Call collect does not scale
A default configuration of call collect (client initiated reporting) does not scale above 50-100 hosts in 3.6 due to clients waiting too long to deliver the reports which fills up the queue. As a workaround, collect_window can be increased to get proporional scale increase; e.g. increasing to 20 from the default of 10 will double the scale.
If you have this issue in 3.6, please increase collect_window and upgrade to 3.6.5 or later which improves stability of clients when they do not regularly get reports collected.
This issue is fixed in 3.7, where call collect scales to thousands of hosts with default configuration.
Installation and Configuration
Installation
There are several steps to bring up a CFEngine installation within an organization:
- Prepare all appropriate machines for installation.
- Configure your network and security.
- Download the CFEngine software.
- Install CFEngine on the Policy Server(s).
- Bootstrap the Policy Server to itself.
- Initiate post-install configuration on the Policy Server.
- Install CFEngine on the Host machine(s).
- Bootstrap the Host(s) to a Policy Server.
See General Installation for a more detailed guide for how to install CFEngine, and links to installation guides for various versions of CFEngine and different configurations.
See Secure Bootstrap for a guide on bootstrapping CFEngine in untrusted networks.
See also: Pre-Installation Checklist, Supported Platforms and Versions
Setup & Configuration
Additional options for configuring CFEngine policy are as follows:
Controlling Frequency Learn how to control frequency settings for verifying CFEngine policy.
Version Control Learn how to put your CFEngine policies under version control.
The Policy Framework Learn what options are available out of the box in CFEngine to configure its masterfiles operation.
Pre-Installation Checklist
Download Packages
System requirements
Please see Installing Enterprise for Production for hardware and configuration requirements, and for Supported Platforms and Versions operating system support.
Required Knowledge
- Linux
- SSH
- bash
- command line text editing (e.g. vi/vim, Emacs)
See Also: Quick-Start Guide to Using vi, Quick-Start Guide to Using PuTTY
Quick-Start Guide to Using vi
Quick-Start Guide To Using vi
This guide is designed for the novice user of CFEngine tutorials—and will introduce the basic use of a powerful tool that is referenced in the CFEngine learning documentation: the vi visual editor.
What is a visual editor? It lets you see multiple lines of the document you are editing—rather than simply issuing commands in the shell prompt. This means you can insert a very large piece of text and navigate anywhere in that text, and make changes.
The vi editor was developed for unix—but can run from any shell prompt, like PuTTY on the Windows platform, and also the Mac. So whatever the user's platform may be, learning to use vi will be very useful in working through the CFEngine tutorials.
When working in the CFEngine tutorials, vi will be used to do things like open files, insert text, save files, and many other functions.
vi will also be used when the CFEngine user starts to actually use the CFEngine software—for things like writing and deploying promises, the core of the CFEngine technology.
Learning the basics of vi is quite simple. The best way is by walking through an example.
Step 1. Inside the shell prompt, simply type “vi”. This will allow the user to insert text and create a new file.
Step 2. type “i” then press the “Enter” key. This takes the user to the insert mode, and allow typing in text or copying and pasting.
Step 3. Type some text—for example, the obligatory “Hello World” (which will be the subject of a later tutorial). Now press "Enter" to go to the next line and type “My name is Gary, and it's nice to meet you.”
The output will look like this:
Hello World
My name is Gary, and it's nice to meet you
Step 4. Now exit the insert mode and go back to command mode by pressing the “esc” key.
Step 5. Save the file by typing “:w (filename)”
Step 6. exit vi by typing “:q”
You can also save and exit with one command, “:wq”
It is important to remember that there are two basic operation modes in vi: the command mode, with which the user opens, saves and exits from files, and the insert mode with which the user inserts text—either by typing it in, or by copying and pasting—and can then edit any part of the text in the file.
open file using vi filename
Quick-Start Guide to Using PuTTY
- Using PuTTY in Simple Steps
- Accessing AWS Virtual Machines via SSH on Windows Using PuTTY and PuTTYgen
Using PuTTY in Simple Steps
This guide is intended for Windows users who are not accustomed to using SSH, or need some additional support for understanding how to work with SSH from their machine (e.g. challenges with key pairs).
It describes how to start using the free, open-source program PuTTY, to securely connect a client computer to a remote Linux/Unix server.
Many of the tutorials to follow will refer to using PuTTY, which is a popular SSH client for Windows workstations.
The important thing about PuTTY is that it is a secure way to connect a client to a server, using the SSH network protocol. It has a powerful and easy-to-use graphical user interface (GUI) and is used to run a remote session over a network.
What is SSH? It is short-form for “Secure Shell,” which means it creates a secure channel over an insecure network—like the internet, for example.
How does SSH do this? By encrypting the communications between the client and the server, using public-key cryptography, which means that a key-pair is generated—one of them public, and the other private, or secret, known only to the user.
Since CFEngine is a client-server enterprise software system, it is essential to access the servers securely. This is true whether the CFEngine system is run on a cloud platform, like Amazon Web Services and many others—or on a private network.
That is where PuTTY comes into the picture, since it uses SSH protocol for connecting a client to a server.
The PuTTY software consists of two separate programs PuTTY and PuTTYgen: They can be downloaded at http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
PuTTYgen is used to generate the encryption key pair while PuTTY, a command-line interface, is used to securely access the CFEngine server, or hub, from a remote client machine, which is called a host in CFEngine terminology.
PuTTYgen is used only when setting up a new client machine on the CFEngine hub. The CFEngine hub will already have an encrypted key-pair that was created when setting up the hub. (See the tutorial, Installing CFEngine on RHEL Using AWS)
The following steps describe how to get the client machine, up and running using PuTTYgen and PuTTY. There are two distinct steps to this process:
Step 1. Use PuTTYgen to create an encrypted key-pair in the .ppk file format that PuTTY uses.
(It is important to note that the key-pair on the hub will probably be in a file format that is different from the PuTTYgen .ppk file format. For example on Amazon Web Services (AWS) and many other cloud computing services, the key-pair file format created when setting up the server (_hub_) will be in the .pem file format.)
Step 2. Configure the PuTTY application in order to securely access the CFEngine hub.
Step 1. consists of the following sequence: First, launch PuTTYgen by double-clicking on the puTTygen icon in the Windows programs menu tree; (It should be inside the PuTTY folder that was created when the PuTTY was downloaded and installed.)
Next, download the key-pair and save it on the local hard disk in the .ppk file format.
a. Click Load. The following Load private key window will pop up:
b. In the Load private key window select All Files (*.*) in the drop down menu next to the File name input box.
c. Navigate to the location on disk where the public-key file was downloaded in earlier steps, in this case a .pem file. Click Open. The following window will appear:
d. Enter a Passphrase and confirm the Passphrase. If no Passphrase is desired, leave those fields empty.
e. When the key has been loaded click the Save private key button.
f. If saving without a Passphrase a dialog box will pop up; click yes to save the key without a Passphrase
g. Now close PuTTYgen.
Accessing AWS Virtual Machines via SSH on Windows Using PuTTY and PuTTYgen
Get PuTTY and PuTTYgen
- To get PuTTY and PuTTYgen, first go to
http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html and either:
- Download and install using the PuTTY binaries installer
- Or, download PuTTY and PuTTYgen individually
Prepare Private Key Using PuTTYgen
- After the binaries have been downloaded and/or installed either:
- Double click
puttygen.exe
from the download location, if downloaded directly. - Or, if the PuTTY installer was used above, one of either:
- Press the
Windows
key +R
key and then typeputtygen
in the field namedOpen
. Then press theEnter
key or clickOK
. - Alternatively, double click puttygen.exe under
C:\Program Files (x86)\PuTTY
(when using Windows 64 bit) orC:\Program Files\PuTTY
(when using Windows 32 bit).
- Double click
The Puttygen Interface. You will load the .pem file that you created in AWS.
- On the PuTTYgen interface click the load button.
- In the Load private key window select All Files (*.*) in the drop down menu next to the File name input box.
- Navigate to the location on disk where the .pem file was downloaded in earlier steps.
- When the key has been loaded click the Save private key button.
- When prompted with a warning about saving without a passphrase, click yes.
The Puttygen popup window. Click Yes
, to proceed without a passphrase. You can also protect your private key with a passphrase that you enter into Key Passprhase
and Confirm Key Passphrase
.
- Finally, navigate to a good location on disk to save the key file, enter a name for the private key, ensure PuTTY Private Key Files (*.ppk) type is selected, and then click the Save button.
- You can now close the Puttygen application. You will call up the .ppk file when you configure the virtual machines using PuTTY.
Configure PuTTY
- Before configuring PuTTY, go back to your AWS Console, then navigate to INSTANCES > Instances.
- Make a note of the 2 different Public DNS entries for the virtual machines that were setup earlier (e.g. ec2xxxxxxxxxxxx.uswest1.compute.amazonaws.com, where the x's represent numbers).
- Launch PuTTY by either:
- Double clicking
putty.exe
from the download location, if downloaded directly. - Or, if the PuTTY installer was used above, one of either:
- Press the
Windows
key +R
key and then typeputty
in the field namedOpen
. Then press theEnter
key or clickOK
. - Alternatively, double click
putty.exe
underC:\Program Files (x86)\PuTTY
(when using Windows 64 bit) orC:\Program Files\PuTTY
(when using Windows 32 bit). - On the PuTTY interface, select
Category > Session
on the left side navigation tree:
- Double clicking
The Putty interface, with Session
selected on the left-side navigation tree.
- Now, we will configure the Putty application, which we will use to set up the two AWS virtual machines.
- The first step is to create a Host Name for the first VM.
- The Host Name consists mainly of the public DNS entry that was created for one of the two virtual machines in AWS. But the DNS is preceded by a user name,
ec2-user
, followed by the@
symbol, which is then followed by the DNS entry.
Setting up the PuTTY configuration with the Host Name, and a Saved Sessions Name.
- Port should be set to
22
. - Connection type should be set to
SSH
. Saved Sessions
can be any label.
Once we have entered our Host Name and our Saved Sessions name, we take the following steps:
* Select Connection > SSH > Auth
on the left side navigation tree.
* Click the Browse
button to select the Private key for authentication
.
* In the Select private key file
window, navigate to the .ppk private key file created earlier, and double-click on it to enter it into PuTTY. Your PuTTY screen should look like this:
Note that Auth
has been selected on left-side tree, in order to bring up this screen.
- Now we go back and select
Category > Session
on the left side navigation tree and then press theSave
button. - Repeat the steps for the second virtual machine, starting from setting the Host Name through pressing the Save button (as described above). Your PuTTY screen should show the two saved virtual machines, which are here named
Examples 1 and 2.
- Note: It may be necessary to redo the steps from selecting
Connection > SSH > Auth
through selecting the .ppk private key file. In other words, when configuring the connection the private key file may not be persistently saved. - Wait a moment, and select
Yes
if prompted. - This prompt will generally only be necessary when trying to login for the very first time.
The PuTTY interface with the two virtual machines saved. We can now proceed to configure those virtual machines with CFEngine.
Login to Virtual Machines Using PuTTY
- If one of the two virtual machines is configured and its details loaded in the PuTTY interface, first select the machine, then click the Open button. This will close the above PuTTY interface and open a command-line window, from which we will setup CFEngine on each of the two machines. One machine will act as the Server and the other as the client, and they will each be set up with different software.
- Once the first virtual machine is logged into, right click the top of PuTTY's application window (e.g. the part of the window decoration displaying the virtual machine name).
- In the contextual menu that then shows click New Session.
- Select the second virtual machine entry in the Saved Sessions list.
- Click Load and then Open.
- Both virtual machines should now be accessed in two different PuTTY command-line windows. Below is an example of what the command-line window will look like.
The PuTTY command-line window, which we will use to configure the virtual machines with CFEngine.
General Installation
There are several steps to bring up a CFEngine installation within an organization:
- Prepare all appropriate machines for installation.
- Configure your network and security.
- Download the CFEngine software.
- Install CFEngine on the Policy Server(s).
- Bootstrap the Policy Server to itself.
- Initiate post-install configuration on the Policy Server.
- Install CFEngine on the Host machine(s).
- Bootstrap the Host(s) to a Policy Server.
Before Installation
Check the Pre-Installation Checklist and Supported Platforms and Versions for requirements and other information that is useful for the installation procedure.
Install Packages
CFEngine Enterprise is provided in two packages; one is for the Policy Server (hub) and the other is for each Host (client).
Note: See Installing Community for the community version of CFEngine)
Log in as root and then follow these steps to install CFEngine Enterprise:
On the designated Policy Server, install the
cfengine-nova-hub
package:[RedHat/CentOS/SUSE] $ rpm -i <server hub package>.rpm [Debian/Ubuntu] $ dpkg -i <server hub package>.deb
On each Host, install the
cfengine-nova
package:[RedHat/CentOS/SUSE] $ rpm -i <agent package>.rpm [Debian/Ubuntu] $ dpkg -i <agent package>.deb
Note: Install actions logged to /var/logs/cfengine-install.log
.
Bootstrap
Bootstrapping a client means to configure it initially. With CFEngine, the default bootstrap:
- records the server's address (accessible as
sys.policy_hub
) and public key, and gives the server the client's key to establish trust (see Bootstrapping) - copies all the contents of
/var/cfengine/masterfiles
on the policy server (AKAsys.masterdir
) to/var/cfengine/inputs
(AKAsys.inputdir
). Seeupdate.cf
for details.
Run the bootstrap command, first on the policy server:
Find the IP address of your Policy Server:
$ ifconfig
Run the bootstrap command:
$ sudo /var/cfengine/bin/cf-agent --bootstrap <IP address of policy server>
The bootstrap command must then be run on any client attaching itself to this server, using the ip address of the policy server (i.e. exactly the same as the command run on the policy server itself).
Post-Installation Configuration
CFEngine itself is configured through policy as well (see Components and Common Control and
The Policy Framework for details). The following basic changes to the default policy will configure
cf-serverd
and cf-execd
for your environment.
def.cf
Find the following line:
"domain" string => "your.domain.here",
Change your.domain.here to your domain name, e.g. example.com.
controls/cf_execd.cf
Find the following line:
mailto => "some-admin-list@me.local";
Change some-admin-list@me.local to your email address.
Note: On some systems this modification should hopefully work without needing to make any additional changes elsewhere. However, any emails sent from the system might also end up flagged as spam and sent directly to a user's junk mailbox.
Server IP Address and Hostname
Edit /etc/hosts
and add an entry for the IP address and hostname of the server.
CFEngine Enterprise Post-Installation Setup
See: What steps should I take after installing CFEngine Enterprise?
More Detailed Installation Guides
Although most install procedures follow the same general workflow, there are several ways of installing CFEngine depending on your environment and which version of CFEngine you are using.
- Installing Enterprise for Production
- Install and test the latest version using our native version, for free!
- Installing CFEngine on virtual machine instances using Amazon Web Services' (AWS) EC2 service
- This is especially useful for people running Windows on their workstation or laptop.
- Install and test the latest version using our pre-packaged Vagrant environment
- Installing CFEngine Community Edition
Next Steps
- Learn about Writing and Serving Policy
Using Amazon Web Services
This guide describes how to install CFEngine on two Red Hat® Enterprise Linux® (RHEL) virtual machines using Amazon Web Services™ (AWS) and SSH. At the time of writing, under certain conditions, setting up an AWS account and using micro-instances is free.
One of the two machines will be a policy server
, while the other will be a host
.
Although these instructions walk through the steps needed to install CFEngine Enterprise on two machines, up to 25 machines can be set up using the same procedure and scripts.
This tutorial will cover the following steps:
- Initial Configuration of the AWS Virtual Machines.
- Configuring the Security Group.
- Configuring SSH Access to the Virtual Machines Using PuTTY (for Windows machines).
- Configuring the Firewall on the Policy Server.
- Installing CFEngine on both the Policy Server and Host Virtual Machines.
Initial Configuration of the Virtual Machines in AWS
Configure 2 RHEL Virtual Machine Instances in AWS
- Login to AWS.
- Under
Create Instance
click onLaunch Instance
. - On the line
Red Hat Enterprise Linux 64 Bit Free tier eligible
press theSelect
button. - On the
Choose Instance Type
screen ensure theMicro Instances
tab on the left is selected.
Configure Instance Details
- Press
Next: Configure Instance Details
. - On the
Configure Instance Details
screen change the number of instances to 2. - Leave
Network
as the default. Subnet
can beNo preference
.- Ensure
Public IP
is checked. - Leave all else at their default values.
Review and Launch
- Click
Review and Launch
. - Make a note of
Security group
name on theReview Instance Launch
screen. - Click
Launch
. - Select
Create a new key
pair in the first drop down menu. - Enter anything as the
Key pair name
. - Click the
Download Key Pair
button and save the .pem file to your local computer. - After the .pem file is saved click the
Launch Instance
button. - On the
Launch Status
screen click theView Instances
button.
Configure the Security Group
- On the left hand side of the AWS console click
NETWORK & SECURITY > Security Groups
- Remembering the
Security group
name from earlier, click on the appropriate line item in the list. - Below the list of security group names will display details for the current security group.
- Click the
Inbound
tab. - Click "Edit" button. A popup window will appear with "SSH" rule already present.
- Click the
+Add Rule
button. SelectHTTP
from the drop-down list. Click "Add Rule" button again. - Select
Custom TCP rule
and enter5308
in thePort range
text entry. Select "Custom IP" from the drop-down menu in the "Source" column. - Copy the "Group ID" from the line containing your "Group Name" and copy the "Group ID" into the text entry in the last column. Click "Save."
- Click the "Edit" button again. On the "Custom TCP" Rule, select "Anywhere" from the "Source" drop-down list. Click "Save."
Accessing the Virtual Machines Using SSH
See: Quick-Start Guide to Using PuTTY
Install and Configure the Firewall
Install the Firewall
- Ensure you are logged into both virtual machines.
- In both enter
sudo yum install system-config-firewall
to install. - Hit 'y' if prompted.
Configure the Firewall on the Policy Server (AKA hub)
The following steps are only necessary for one of the two virtual machines, the one that is designated as the policy server; these steps can be omitted on the second (client machine). Note that CFEngine refers to a client machine by the name Host
:
- When system-config-firewall is installed, enter
sudo system-config-firewall
- In the
Firewall Configuration
screen use theTab
key to go to Customize. - Hit the
Enter
key. Below is theFirewall Configuration
window that comes up:
Open Port 80 (HTTPD)
- On the
Trusted Services
screen, scroll down toWWW (HTTP)
, AKA port 80. - Hit the
Space Bar
to toggle theWWW
entry (i.e. ensure it is on, showing an asterisk beside the name).
Open Port 5308 (CFEngine)
- Hit the
Tab
key again untilForward
is highlighted, then hitEnter
. - Hit the
Tab
key untilAdd
is highlighted, then hitEnter
. - Enter
5308
in thePort
section. - Hit the
Tab
key and entertcp
in theProtocol
section. - Hit the
Tab
key until OK is highlighted, and hitEnter
.
The Port and Protocol
are entered in the blue boxes, with entries of 5308
and tcp
respectively.
Then the Tab
key is used to highlight the OK
button, and the user presses Enter
.
Wrapping Up Firewall Configuration
- Hit the
Tab
key untilClose
is highlighted, and hitEnter
. - Hit the
Tab
key or arrow keys untilOK
is highlighted, and hitEnter
.
Disabling Firewall on a Host (Warning: Only Do This If Absolutely Necessary)
For the second virtual machine, which is the client machine (also called host
), you may need to do the following if you see an error when bootstrapping this virtual machine in later steps:
* In the Firewall Configuration
screen use the Tab
key to go to Firewall.
* Turn off the firewall by toggling the entry with the Space
bar.
Note: Turning off the firewall in a production environment is considered unsafe.
CFEngine Installation Overview
We ready now ready to install the CFEngine software on both the server and client virtual machines. These also referred to as the “hub” and “host” machines, respectively. During the course of the instructions outlined in this guide, you will perform the following tasks:
- Install CFEngine Enterprise onto a Policy Server and onto Hosts. A Policy Server (hub) is a CFEngine instance that contains promises (business policy) that get deployed to Hosts. Hosts are clients that retrieve and execute promises.
- Bootstrap the Policy Server to itself and then bootstrap each of the Hosts to the Policy Server. Bootstrapping establishes a trust relationship between the Policy Server and all Hosts. Thus, business policy that you create in the Policy Server can be deployed to Hosts throughout your company. Bootstrapping completes the installation process.
- Log in to the Mission Portal. The Mission Portal is a graphical user interface that allows you to verify the actual state of all your Hosts, thus ensuring that your promises are being executed. By using the Design Center inside the Mission Portal, you can also define new desired states (business policies) for your infrastructure.
- Try out the Tutorials. Links to three tutorials give you a head start on learning CFEngine.
Step 1. Download and install Enterprise on a Policy Server
Run the following script on your designated Policy Server (hub), the virtual machine with the configured firewall from earlier steps:
$ wget http://cfengine.package-repos.s3.amazonaws.com/quickinstall/quick-install-cfengine-enterprise.sh && sudo bash ./quick-install-cfengine-enterprise.sh hub
This script installs the latest CFEngine Enterprise Policy Server on your server machine.
Step 2. Bootstrap the Policy Server
- The Policy Server must be bootstrapped to itself. Find the IP address of your Policy Server:
$ ifconfig
. Run the bootstrap command:
sudo /var/cfengine/bin/cf-agent --bootstrap <IP address of policy server>
Example:
$ sudo /var/cfengine/bin/cf-agent --bootstrap 172.31.3.25
Upon successful completion, a confirmation message appears: "Bootstrap to '172.31.3.25' completed successfully!"
Type the following to check which version of CFEngine your are running:
/var/cfengine/bin/cf-promises --version
The Policy Server is now installed.
Step 3. Install Enterprise on Host (Client)
- Ensure you are logged into the host machine setup earlier.
- Install CFEngine client version using the following:
$ wget http://cfengine.package-repos.s3.amazonaws.com/quickinstall/quick-install-cfengine-enterprise.sh && sudo bash ./quick-install-cfengine-enterprise.sh agent
Note: The installation will work on 64-bit and 32-bit client machines (the host requires a 64-bit machine).
The client software (host), has been installed on the second virtual machine.
Note: You can install CFEngine Enterprise on up to 25 hosts using the script above.
Step 4. Bootstrap the Host to the Policy Server
- All hosts must be bootstrapped to the Policy Server in order to establish a connection between the
Host
and thePolicy Server
. Run the same commands that you ran in Step 2,
$ sudo /var/cfengine/bin/cfagent bootstrap <IP address of policy server>
.Example:
$ sudo /var/cfengine/bin/cfagent bootstrap 172.31.3.25
The installation process is complete and CFEngine Enterprise is up and running on your system.
Step 5. Log in to the Mission Portal
- The Mission Portal is immediately accessible. Connect to the Policy Server through your web browser at: http://
(Note: The External IP address is available in the AWS console). - The default username for the Mission Portal is
admin
, and the password is alsoadmin
. - The Mission Portal runs TCP port 80 by default. Configure mission portal to use HTTPS instead of HTTP.
- During the initial setup, the Host(s) might take a few minutes to show up in the Mission Portal. Refresh the web page and login again if necessary.
What Next?
Tutorials
Configure and deploy a policy using sketches in the Design Center.
This tutorial teaches you how to configure and deploy business policy by using the Design Center application in the Mission Portal. Next, it shows you how to verify that your business policy is being activated by viewing the Reports in the Mission Portal.
Distribute files from a central location.
Whereas the first tutorial in this list teaches you how to deploy business policy through the Mission Portal, this advanced, command-line tutorial shows you how to distribute policy files from the Policy Server to all pertinent Hosts.
Recommended Reading
Installing Enterprise 25 Free
These instructions describe how to install the latest version of CFEngine Enterprise 25 Free. This is the full version of CFEngine Enterprise, but the number of Hosts (clients) is limited to 25.
Note the following requirements:
- To install this version of CFEngine Enterprise, your machine must be running a recent version of Linux. This installation script has been tested on RHEL 5 and 6, SLES 11, CentOS 5 and 6, and Debian 6 and 7.
- You need a minimum of 2 GB of available memory and a modern 64 bit processor.
- Plan for approximately 100MB of disk space per host. You should provide an extra 2G to 4G of disk space if you plan to bootstrap more hosts later.
- You need a least two VMs/servers, one for the Policy Server and one for a Host (client). They must be on the same network.
- The Policy Server needs to run on a dedicated OS with a vanilla installation (i.e. it only has repositories and packages officially supported by the OS vendor)
Installation Overview
During the course of the instructions outlined in this guide, you will perform the following tasks:
- Install CFEngine Enterprise onto a Policy Server and onto Hosts. A Policy Server (hub) is a CFEngine instance that contains promises (business policy) that get deployed to Hosts. Hosts are clients that retrieve and execute promises.
- Bootstrap the Policy Server to itself and then bootstrap each of the Hosts to the Policy Server. Bootstrapping establishes a trust relationship between the Policy Server and all Hosts. Thus, business policy that you create in the Policy Server can be deployed to Hosts throughout your company. Bootstrapping completes the installation process.
- Log in to the Mission Portal. The Mission Portal is a graphical user interface that allows you to verify the the actual state of all your Hosts, thus ensuring that your promises are being executed. By using the Design Center inside the Mission Portal, you can also define new desired states (business policies) for your infrastructure.
- Try out the Tutorials. Links to three tutorials give you a head start on learning CFEngine.
1. Download and install Enterprise on a Policy Server
Please Note: Internet access is required from the host if you wish to use the quick install script.
Run the following script on your designated Policy Server (hub) 64-bit machine (32-bit is not supported on the Policy Server):
$ wget http://cfengine.package-repos.s3.amazonaws.com/quickinstall/quick-install-cfengine-enterprise.sh && sudo bash ./quick-install-cfengine-enterprise.sh hub
This script installs the latest CFEngine Enterprise Policy Server on your machine.
2. Bootstrap the Policy Server
The Policy Server must be bootstrapped to itself. Find the IP address of your Policy Server (type $ ifconfig).
Run the bootstrap command:
$ sudo /var/cfengine/bin/cf-agent --bootstrap <IP address of policy server>
Example: $ sudo /var/cfengine/bin/cf-agent --bootstrap 192.168.1.12
Upon successful completion, a confirmation message appears: "Bootstrap to '192.168.1.12' completed successfully!"
Type the following to check which version of CFEngine your are running:
$ /var/cfengine/bin/cf-promises --version
The Policy Server is installed.
3. Install Enterprise on Hosts
Install Enterprise on your designated Host(s) by running the script below. Per the Free 25 agreement, you can install Enterprise on 25 Hosts. Note that the Hosts must be on the same network as the Policy Server that you just installed in Step 2.
$ wget http://cfengine.package-repos.s3.amazonaws.com/quickinstall/quick-install-cfengine-enterprise.sh && sudo bash ./quick-install-cfengine-enterprise.sh agent
Note that this installation works on 64- and 32-bit machines.
4. Bootstrap the Host to the Policy Server
All Hosts must be bootstrapped to the Policy Server in order to establish a connection between the Host and the Policy Server. Run the same commands that you ran in Step 3.
$ sudo /var/cfengine/bin/cf-agent --bootstrap <IP address of policy server>
Example: $ sudo /var/cfengine/bin/cf-agent --bootstrap 192.168.1.12
The installation process is complete and CFEngine Enterprise is up and running on your system.
5. Log in to the Mission Portal
The Mission Portal is immediately accessible. Connect to the Policy Server through your web browser at:
http://<IP address of your Policy Server>
username: admin password: admin
The Mission Portal runs TCP port 80 by default. (Click here to configure the Mission Portal to use HTTPS instead of HTTP.) During the initial setup, the Host(s) might take a few minutes to show up in the Mission Portal. Simply refresh the web page and login again if necessary.
Note: If you are running Enterprise with Vagrant, you must add the
correct port: http://localhost:
Tutorials
Configure and deploy a policy using sketches in the Design Center.
This tutorial teaches you how to configure and deploy business policy by using the Design Center application in the Mission Portal. Next, it shows you how to verify that your business policy is being activated by viewing the Reports in the Mission Portal.
Distribute files from a central location.
Whereas the first tutorial in this list teaches you how to deploy business policy through the Mission Portal, this advanced, command-line tutorial shows you how to distribute policy files from the Policy Server to all pertinent Hosts.
Recommended Reading
Rate your experience
Everyone is a first-time user a some point. We want to make the CFEngine Enterprise installation process easy for all of our new users. Before you forget your first-time experience, we would love for you to let us know how we can improve on this process.
Using Vagrant
The CFEngine Enterprise Vagrant Environment provides an easy way to test and explore CFEngine Enterprise. This guide describes how to set up a client-server model with CFEngine and, through policy, manage both machines. Vagrant will create one VirtualBox VM to be the Policy Server (server), and another machine that will be the Host Agent (client), or host that can be managed by CFEngine. Both will will run CentOS 6.5 64-bit and communicate on a host-only network. Apart from a one-time download of Vagrant and VirtualBox, this setup requires just one command and takes between 5 and 15 minutes to complete (determined by your Internet connection and disk speed). Upon completion, you are ready to start working with CFEngine.
Requirements
- 2G disk space
- 1G memory
- CPU with VT extensions capable of running 64bit guests
Note: VirtualBox requires that your computer support hardware virtualization in order to make use of the CentOS 64-bit virtual machines mentioned above. This is sometimes turned on or off in BIOS settings, but not all processors and motherboards necessarily support hardware virtualization.
If your system lacks this support you will need to choose another computer to take advantage of the 64-bit virtual machines or install CFEngine using a different approach.
Overview
- Install Vagrant
- Install Virtualbox
- Start the CFEngine Enterprise Vagrant Environment
- Log in to the Mission Portal
- Stop CFEngine Enterprise
- Uninstall
Install Vagrant
This tutorial uses Vagrant to configure your VMs. It is available for Linux, Windows and MacOS and can be downloaded from vagrantup.com (this guide has been tested with version 1.7.4). After downloading Vagrant, install it on your computer. You may want to reference the Windows Mac or Linux vagrant install guides.
Install Virtualbox
This tutorial uses VirtualBox to create virtual machines on your computer, to which Vagrant deploys CFEngine. VirtualBox can be downloaded from virtualbox.org (this guide has been tested with version 5.0.16). After downloading VirtualBox, install it on your computer.
Note: To avoid problems, disable other virtualization environments you are running.
Start the CFEngine Enterprise 3.6 Vagrant Environment
Step 1. Download our ready-made Vagrant project tar-file.
Step 2. Save and unpack the file anywhere on your drive; this creates a Vagrant Project directory.
Step 3. Open a terminal and navigate to the Vagrant Project directory (e.g.
/home/user/CFEngine_Enterprise_vagrant_quickstart-3.7.7-1
, or C:\CFEngine_Enterprise_vagrant_quickstart-3.7.7-1
) and enter the following command:
$ vagrant up
Vagrant performs the following processes:
- Downloads the CentOS basebox used for both the hub and the client (if it has not already been cached by vagrant.
- Provisions, installs and bootstraps the hub
- Provisions, installs and bootstraps clients
The basebox is ~500MB.
Note: If you want to use more hosts in this environment, you can edit the Vagrantfile text file in the directory that you have just created. Change the line that says "hosts = 1" to the number of hosts that you want in the setup. The maximum supported in this evaluation version of CFEngine is 25.
Log in to the Mission Portal
At the end of the setup process, you can use your browser to log in to the Mission Portal:
http://localhost:9002
username: admin
password: admin
Note: It may take up to 15 minutes before the hosts register in Mission Portal.
That's all there is to it, the install is complete! Move on and explore the environment.
Exploring the Environment
Accessing VMs
Accessing via SSH
The standard vagrant ssh key is configured. To ssh to a host run vagrant ssh
myhost
where myhost
is the name of a running vm as seen in the vagrant
status
output. Both the 'root' and 'vagrant' users passwords are set to
'vagrant'.
Example:
$ vagrant ssh hub
Last login: Fri Jun 13 18:58:10 2014 from 10.0.2.2
Accessing via GUI
If you launch the virtualbox GUI you should find the vagrant vms named
CFEngine Enterprise 3.7.7-1 hub
, and CFEngine Enterprise 3.7.7-1 agent host001
. Additionally, you can uncomment the v.gui=true
option in the Vagrantfile
to have the console gui start with the vms.
Note: There are two v.gui
settings to uncomment; one for the hub, and one
for the clients.
Check the status of the vms
Running vagrant status
from the vagrant project directroy will produce
output like this.
$ vagrant status
Current machine states:
hub not created (virtualbox)
host001 not created (virtualbox)
This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
Start or resume the environment
To start or resume a halted environment simply run vagrant up
from within the
vagrant project directory.
$ vagrant up
Stop the environment (Halt/Suspend/Destroy)
To shut down the vms run vagrant halt
. This will preserve the vms and any
changes made inside.
$ vagrant suspend
==> hub: Saving VM state and suspending execution...
==> host001: Saving VM state and suspending execution...
To suspend the vms run vagrant suspend
. This will freeze the state of each vm
and allows for latter resuming of the environment.
$ vagrant halt
==> host001: Attempting graceful shutdown of VM...
==> hub: Attempting graceful shutdown of VM...
At any time you can run vagrant destroy
to remove the provisioned vms. This will
delete the vms and any modifications made to the environment will be lost.
$ vagrant destroy
host001: Are you sure you want to destroy the 'host001' VM? [y/N] y
==> host001: Forcing shutdown of VM...
==> host001: Destroying VM and associated drives...
==> host001: Running cleanup tasks for 'shell' provisioner...
==> host001: Running cleanup tasks for 'shell' provisioner...
==> host001: Running cleanup tasks for 'shell' provisioner...
hub: Are you sure you want to destroy the 'hub' VM? [y/N] y
==> hub: Forcing shutdown of VM...
==> hub: Destroying VM and associated drives...
==> hub: Running cleanup tasks for 'shell' provisioner...
==> hub: Running cleanup tasks for 'shell' provisioner...
==> hub: Running cleanup tasks for 'shell' provisioner...
Uninstall Vagrant Environment
When you have completed your evaluation are ready to use CFEngine on production servers, remove the VMs that you created above by following these simple instructions:
To remove the VMs entirely, type: vagrant destroy
If you are completely done and do not anticipate using them anymore, you can
also remove the base box centos-6.5-x86_64-cfengine_enterprise-3.6.6-1
that was
downloaded. You can see it by typing vagrant box list
. To delete the basebox
run vagrant box remove centos-6.5-x86_64-cfengine_enterprise-3.6.6-1 virtualbox
.
Note: Running vagrant up
from the vagrant project directory again will
re-download this basebox.
Vagrant and VirtualBox are useful general purpose programs, so you might want to keep them around. If not, follow the standard procedures for your OS to remove these applications.
Next Steps
See also
Installing Enterprise for Production
These instructions describe how to install the latest version of CFEngine Enterprise in a production environment using pre-compiled rpm and deb packages for Ubuntu, Debian, Redhat, CentOS, and SUSE.
General Requirements
CFEngine recommends the following:
Host Memory
During normal operation the CFEngine processes consume about 30 MB of resident memory (RSS) on hosts with the agent only (not acting as Policy Server).
However there might be spikes due to e.g. commands executed from the CFEngine policy so it is generally recommended to have at least 256 MB available memory in order to run the CFEngine agent software.
Host disk
On Unix-like systems, CFEngine can consume up to 100 MB of the partition mounted where CFEngine is installed (usually /var/cfengine). On Windows systems, CFEngine can consume up to 1GB (usually C:\Program Files\Cfengine). The higher disk usage on Windows is due to lack of support for sparse files on this platform, which is utilized by a dependency of CFEngine (lmdb) when available.
Maximum disk space used by diff reports (contexts, variables, software installed, software patches, lastseen hosts and promise executions) can be set as max_client_history_size variable located in masterfiles/def.cf.
Network
Verify that the machine’s network connection is working and that port 5308 (used by CFEngine) is open for both incoming and outgoing connections.
If a firewall is active on your operating system, adapt it to it to allow for communication on port 5308 or disable it.
CFEngine bundles all critical dependencies into the package; therefore, additional software is not required.
Requirements for VIOS
CFEngine Enterprise has Virtual I/O Server (VIOS) Recognized status from IBM. This means that CFEngine Enterprise has been technically verified by IBM to be installed in and manage VIOS environments.
During testing, CFEngine Enterprise was seen to use up to 2% of the VIOS CPU
during cf-agent
runs with the default CFEngine policy. The resource
utilization may vary depending on the policy CFEngine is running.
The VIOS should be configured with Shared Processors in Uncapped mode.
Policy Server Requirements
cfapache and cfpostgres users
The CFEngine Server requires two users: cfapache and cfpostgres. If these users do not exist during installation of the server package, they will be created, so if there are constraints on user creation, please ensure that these users exists prior to installation.
These users are not required nor created by the agent package.
Dedicated OS
The CFEngine Server is only supported when installed on a dedicated, vanilla OS (i.e. it only has repositories and packages officially supported by the OS vendor). This is because the CFEngine Server uses services, e.g. apache, that are configured for CFEngine and may conflict with other custom application configurations.
One option, especially for smaller installations, is to run the CFEngine Server in a VM. But please consider the performance requirements when doing this.
Disk sizing and partitioning
It is recommended that /var/cfengine/state/pg is mounted on a separate disk. This will give PostgreSQL, which is very disk I/O intensive, dedicated resources.
Plan for approximately 100MB of disk space per bootstrapped agent. This means that, for a server with 5000 hosts, you should have at least 500 GB available on the database partition.
xfs is strongly recommended as the file system type for the file system mounted on /var/cfengine/state/pg. ext4 can be used as an alternative, but ext3 should be avoided.
Disk speed
The disk that serves PostgreSQL (/var/cfengine/state/pg) should be able to perform at least 2000 IOPS (in 16KiB block size). The disk mounted on /var/cfengine should be able to perform at least 500 IOPS. SSD is recommmended for the disk that PostgreSQL writes to.
Note Your storage IOPS specification may be given in 4KiB block size, in which case you would need to divide it by 4 to get the corresponding 16KiB theoretical maximum.
Open file descriptors
The policy server should ideally be able to accept connections from all clients; i.e. to allow at least as many incoming connections as there are clients.
The system limit for this is controlled by ulimit -n
; so the parent process from which you bootstrap should, for a server with 5000 hosts, run ulimit -n 5000
first.
You should also add such a ulimit -n
command to the script that implements service cfengine3 start
(and restart
) and to any policy that starts cf-serverd
or cf-hub
.
For very large numbers of clients, it may be advantageous to build a custom kernel to allow setting ulimit -n
high enough.
You should also amend the value of maxconnections
set in cf_serverd.cf
under /var/cfengine/masterfiles/controls/
to the number of clients, likewise.
Memory
Minimum 2GB memory, but not lower than 8MB per bootstrapped agent. This means that, for a server with 5000 hosts, you should have at least 40GB of memory.
CPU
A modern 64-bit processor with 8 or more cores for handling up to 5000 bootstrapped agents.
Download Packages
Install Packages
CFEngine Enterprise is provided in two packages; one is for the Policy Server (hub) and the other is for each Host (client).
Log in as root and then follow these steps to install CFEngine Enterprise:
On the designated Policy Server, install the
cfengine-nova-hub
package:[RedHat/CentOS/SUSE] # rpm -i <hub package>.rpm [Debian/Ubuntu] # dpkg -i <hub package>.deb
On each Host, install the
cfengine-nova
package:[RedHat/CentOS/SUSE] # rpm -i <agent package>.rpm [Debian/Ubuntu] # dpkg -i <agent package>.deb [Solaris] # pkgadd -d <agent package>.pkg all [AIX] # installp -a -d <agent package>.bff cfengine.cfengine-nova [HP-UX] # swinstall -s <full path to agent package>.depot cfengine-nova
Note: Install actions logged to /var/logs/cfengine-install.log
.
Bootstrap
Run the bootstrap command, first on the policy server and then on each host:
$ /var/cfengine/bin/cf-agent --bootstrap <IP address of the Policy Server>
Licensed installations
If you are evaluating CFEngine Enterprise or otherwise using it in an environment with less than 25 agents connecting to a Policy Server, you do not need a license and there is no expiry.
If you are a customer, please send the Policy Server's public key (/var/cfengine/ppkeys/localhost.pub
) to
CFEngine support to obtain a license. CFEngine will send you a license.dat
file. Copy the obtained license file to
/var/cfengine/masterfiles/license.dat
Next Steps
When bootstrapping is complete, CFEngine is up and running on your system.
The Mission Portal is immediately accessible. Connect to the Policy Server
through your web browser at http://<IP address of your Policy Server>
.
To be able to use the Mission Portal's Design Center front-end, continue with integrating Mission Portal with git.
Learn more about CFEngine by using the following resources:
Tutorial: Tutorial for Running Examples
Tutorial: Configure and deploy a policy using sketches in the Design Center.
Installing Community
These instructions describe how to download and install the latest version of CFEngine Community using pre-compiled rpm and deb packages for Ubuntu, Debian, Redhat, CentOS, and SUSE.
It also provides instructions for the following:
- Install CFEngine on a Policy Server (hub) and on a Host (client). A Policy Server (hub) is a CFEngine instance that contains promises (business policy) that get deployed to Hosts. Hosts are clients that retrieve and execute promises.
- Bootstrap the Policy Server to itself and then bootstrap the Host(s) to the Policy Server. Bootstrapping establishes a trust relationship between the Policy Server and all Hosts. Thus, business policy that you create in the Policy Server can be deployed to Hosts throughout your company. Bootstrapping completes the installation process.
Tutorials, recommended reading. and production environment recommendations appear at the end of this page.
Quick Setup Installation Script
Please Note: Internet access is required from the host if you wish to use the quick install script.
Use the following script to install CFEngine on your 32- or 64-bit machine.
$ wget -O- http://cfengine.package-repos.s3.amazonaws.com/quickinstall/quick-install-cfengine-community.sh | sudo bash
- Run this script on your designated Policy Server machine and on your designated Host machine(s).
- Bootstrap the Policy Server to itself and then bootstrap your Host(s) to the Policy Server by running the following command:
$ sudo /var/cfengine/bin/cf-agent --bootstrap <IP address of policy server>
If you require more details on bootstrapping, review Step 3 below. Bootstrapping completes the installation. - Go to the Tutorials section to learn how to use CFEngine.
1. Download Packages
Select the package to download that matches your operating system. This stores the cfengine-community_3.6.1-1_* file onto your machine.
Redhat/CentOS/SUSE 64-bit:
$ wget http://cfengine.package-repos.s3.amazonaws.com/community_binaries/cfengine-community-3.6.1-1.x86_64.rpm
Redhat/CentOS/SUSE 32-bit:
$ wget http://cfengine.package-repos.s3.amazonaws.com/community_binaries/cfengine-community-3.6.1-1.i386.rpm
Ubuntu/Debian 64-bit:
$ wget http://cfengine.package-repos.s3.amazonaws.com/community_binaries/cfengine-community_3.6.1-1_amd64.deb
Ubuntu/Debian 32-bit:
$ wget http://cfengine.package-repos.s3.amazonaws.com/community_binaries/cfengine-community_3.6.1-1_i386.deb
2. Install CFEngine on a Policy Server
Install the package on a machine designated as a Policy Server. A Policy Server is a CFEngine instance that contains promises (business policy) that get deployed to Hosts. Hosts are instances (clients) that retrieve and execute promises.
Choose the right command for your operating system:
Redhat/CentOS/SUSE 64-bit:
$ sudo rpm -i cfengine-community-3.6.1-1.x86_64.rpm
Redhat/CentOS/SUSE 32-bit:
$ sudo rpm -i cfengine-community_3.6.1-1.i386.rpm
Ubuntu/Debian 64-bit:
$ sudo dpkg -i cfengine-community_3.6.1-1_amd64.deb
Ubuntu/Debian 32-bit:
$ sudo dpkg -i cfengine-community_3.6.1-1_i386.deb
Note: You might get a message like this: "Policy is not found in /var/cfengine/inputs, not starting CFEngine." Do not worry; this is taken care of during the bootstrapping process.
3. Bootstrap the Policy Server
The Policy Server must be bootstrapped to itself. Find the IP address of your Policy Server (type $ ifconfig).
Run the bootstrap command:
$ sudo /var/cfengine/bin/cf-agent --bootstrap <IP address of policy server>
Example: $ sudo /var/cfengine/bin/cf-agent --bootstrap 192.168.1.12
Upon successful completion, a confirmation message appears: "Bootstrap to '192.168.1.12' completed successfully!"
Type the following to check which version of CFEngine your are running:
$ /var/cfengine/bin/cf-promises --version
The Policy Server is installed.
4. Install CFEngine on a Host
As stated earlier, Hosts are instances that retrieve and execute promises from the Policy Server. Install a package on your Host. Use the same package you installed on the Policy Server in Step 2. Note that you must have access to at least one more VM or server and it must be on the same network as the Policy Server that you just installed.
5. Bootstrap the Host to the Policy Server
The Host(s) must be bootstrapped to the Policy Server in order to establish a connection between the Host and the Policy Server. Run the same commands that you ran in Step 3.
$ sudo /var/cfengine/bin/cf-agent --bootstrap <IP address of policy server>
Example: $ sudo /var/cfengine/bin/cf-agent --bootstrap 192.168.1.12
The CFEngine installation process is complete.
Tutorials
Create a policy ("hello world")
Step 1. Create a file called hello_world.cf and add the following content:
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
reports: # This is a promise type.
cfengine_3:: # This means the promise will only
# be kept on a CFEngine_3 system.
"Hello World"; # This is a simple promise; it generates a report
# that says "Hello world".
}
Step 2. Run the policy:
$ sudo /var/cfengine/bin/cf-agent hello_world.cf
The policy displays the following output:
R: Hello world
Find more policy examples in the Examples section.
Create a distributed policy
Create a policy that ensures (promises) that a file called example.txt will always exist on the Host.
Step 1. On the Policy Server, create a file called mypolicy.cf and add it to the /var/cfengine/masterfiles directory:
$ sudo <editor> /var/cfengine/masterfiles/mypolicy.cf
Step 2. Add the following lines to the file:
bundle agent example
{
files:
cfengine_3:: # This is a class context (the promise will only
# be kept on a CFEngine_3 system)
"/home/vagrant/example.txt" # Path and name of the file we wish to ensure exists
create => "true"; # Make sure the file exists; create if it does not
}
Step 3. Update /var/cfengine/masterfiles/promises.cf to include this new policy. To do so, modify the promises.cf file to ensure that (1 the mypolicy.cf file is being included in the next policy distribution and that (2 example is in the bundlesequence.
$ sudo <editor> /var/cfengine/masterfiles/promises.cf
...
bundlesequence => {
# Common bundles first for best practice
"def",
"example",
...
inputs => {
# Global common bundles
"def.cf",
"mypolicy.cf",
...
The process is complete. The next time CFEngine runs on the Host (which by default is every 5 minutes), it will pull down the latest policy update and ensure that the example.txt file exists (this is the desired state). In fact, any Host that has installed CFEngine will contain the example.txt file (because we defined the cfengine_3:: class above).
Try these advanced tutorials:
- Tutorial for Running Examples
- Distribute files from a central location. This advanced, command-line tutorial shows you how to distribute policy files from the Policy Server to all pertinent Hosts.
Recommended Reading
- CFEngine language concepts
Production Environment
If you plan to use Community in a production environment, complete the following general requirements:
Host(s) Memory
256 MB available memory in order to run the CFEngine agent software (cf-agent).
Disk Storage
A full installation of CFEngine requires 25 MB. Additional disk usage depends on your specific policies, especially those that concern reporting.
Network
Verify that the machine’s network connection is working and that port 5308 (used by CFEngine) is open for both incoming and outgoing connections.
If iptables are active on your operating system, stop this service or adapt it to allow for communication on the above ports. If applicable, type the following two commands: /
etc/init.d/iptables stop
andchkconfig iptables off
Rate your experience
Everyone is a first-time user a some point. We want to make the CFEngine Enterprise installation process easy for all of our new users. Before you forget your first-time experience, we would love for you to let us know how we can improve on this process.
Secure Bootstrap
This guide presumes that you already have CFEngine properly installed and running on the policy hub, the machine that distributes the policy to all the clients. It also presumes that CFEngine is installed, but not yet configured, on a number of clients.
We present a step-by-step procedure to securely bootstrapping a number of servers (referred to as clients) to the policy hub, over a possibly unsafe network.
Introduction
CFEngine's trust model is based on the secure exchange of keys. This exchange of keys between client and hub, can either happen manually or automatically. Usually this step is automated as a dead-simple "bootstrap" procedure:
cf-agent --bootstrap $HUB_IP
It is presumed that during this first key exchange, the network is trusted, and no attacker will hijack the connection. After "bootstrapping" is complete, the node can be deployed in the open internet, and all connections are considered secure.
However there are cases where initial CFEngine deployment is happening over an insecure network, for example the Internet. In such cases we already have a secure channel to the clients, usually ssh, and we use this channel to manually establish trust from the hub to the clients and vice-versa.
Manual Trust Establishment
This procedure concerns CFEngine version 3.6 or earlier. While this fully manual procedure should always work, from version 3.7 onwards there is a simpler semi-automatic procedure for establishing trust.
On the policy hub
We must change the policy we're distributing to fully locked-down
settings. So after we have set-up our hub (using the standard procedure
of cf-agent --bootstrap $HUB_IP
) we take care of the following:
Cf-serverd must never accept a connection from a client presenting an untrusted key. So in
body server control
we must empty thetrustkeysfrom
attribute:trustkeysfrom => {};
Since we will be manually bootstrapping the clients, we need to distribute a proper
failsafe.cf
policy. (**NOTE:**failsafe.cf
is a file auto-generated in theinputs
directory when we runcf-agent --bootstrap
).In order to do that, we copy hub's
failsafe.cf
tomasterfiles
directory:cp /var/cfengine/inputs/failsafe.cf /var/cfengine/masterfiles/
We'll edit that copy in
masterfiles
in the next step.All
copy_from
files promises must never connect to an untrusted server, which means that the following line should not be found anywhere:trustkey => "true"
All occurences of
trustkey
inmasterfiles
directory should be changed to "false", or be removed (since it defaults to false anyway). It is certain thatfailsafe.cf
that we copied in the previous step will contain such occurences that should be changed. (Those occurences are the reason that automatic bootstrapping requires a trusted network).The previous changes in
masterfiles
need to be properly propagated to theinputs
directory. The automated way to do that is to run the update.cf policy:cf-agent -f update.cf
Get the hub's key fingerprint, we'll need it later:
HUB_KEY=`cf-key -p /var/cfengine/ppkeys/localhost.pub`
On each client we deploy
We should not follow the automatic method, i.e. the
cf-agent --bootstrap
command. We will perform a
manual bootstrap.
Generate a private/public key pair by running
cf-key
.Get the client's key fingerprint, we'll need it later:
CLIENT_KEY=`cf-key -p /var/cfengine/ppkeys/localhost.pub`
Write the policy hub's IP address to
policy_server.dat
:echo $HUB_IP > /var/cfengine/policy_server.dat
Manually copy the modified
failsafe.cf
from hub'smasterfiles
directory into client'sinputs
directory. You should do it in a secure manner, for example usingscp
with properly trusted fingerprint of the remote host.NOTE: At this step, you can try running the failsafe policy. Because trust between the two hosts has not been established, you will get a failure, which is totally expected and means everything is correct and secure:
$# cf-agent -f failsafe.cf error: TRUST FAILED, server presented untrusted key: MD5=cc27570b8b831192d9f20b54d07dd80b error: No suitable server responded to hail error: TRUST FAILED, server presented untrusted key: MD5=cc27570b8b831192d9f20b54d07dd80b error: No suitable server responded to hail [ ... ]
Put the hub's key into the client's trusted keys:
scp $HUB_IP:/var/cfengine/ppkeys/localhost.pub /var/cfengine/ppkeys/root-${HUB_KEY}.pub
Final steps
Put the client's key into the hub's trusted keys. So on the hub, run:
scp $CLIENT_IP:/var/cfengine/ppkeys/localhost.pub /var/cfengine/ppkeys/root-${CLIENT_KEY}.pub
If you now run the failsafe policy on each and every client, it should succeed:
cf-agent -f failsafe.cf
Congratulations, you have performed a fully manual bootstrap procedure for your clients!
Upgrading to 3.6
This guide documents our recommendation on how to upgrade an existing installation of CFEngine Community 3.4/3.5 and CFEngine Enterprise 3.0/3.5 to CFEngine 3.6.1, as well as upgrades from 3.6.
Our recommendation is to upgrade the Policy Server first. The rationale is that it is normally a dedicated machine with no business-relevant duties, so the risk is lower.
Upgrade masterfiles and Policy Server (3.6.X to 3.6.X+1)
If you are doing a minor-minor 3.6 upgrade (e.g. from 3.6.5 to 3.6.6), the upgrade is easier. We would however still recommend to perform a masterfiles upgrade (ideally in a test environment first) to get all the enhancements and fixes. The masterfiles are available in the hub package and separately on the download page (Community and Enterprise editions share masterfiles as of 3.6).
Normally these new files did not exist in the preivous version or can be completely replaced by the old ones, the only ones that are likely changed by you are def.cf and promises.cf. For these two files, we would need to do a diff between your version and the new version and integrade the diff instead of replacing the whole file.
When the new masterfiles have been created and cf-promises promises.cf and cf-promises update.cf succeeds, you are ready to upgrade the Policy Server. That entails to
- stop the CFEngine services
- upgrade the hub package
- replace /var/cfengine/masterfiles with your new integrated masterfiles
- replace (or merge with your changes) /var/cfengine/state/pg/data/postgresql.conf with /var/cfengine/share/postgresql/postgresql.conf.cfengine to update your database configuration.
- start the CFEngine services
Check the version with /var/cfengine/bin/cf-promises -V, and if you are running Enterprise, the Mission Portal About page.
If your clients get promise failures (not kept) similar to "Can't stat file '/var/cfengine/master_software_updates/cf-upgrade/linux.x86_64/cf-upgrade' on '
If everything looks good, you are ready to upgrade the clients, please skip to Prepare Client upgrade (all versions) followed by Complete Client upgrade (all versions) below.
Prepare masterfiles and the Policy Server for upgrade (3.5 to 3.6)
- Merge your masterfiles with the CFEngine 3.6 policy framework on an infrastructure separate from your existing CFEngine installation.
- Identify existing modifications to the masterfiles directory. If patches from version control are unavailable or require verification, a copy of /var/cfengine/masterfiles from a clean installation of your previous version can help identify changes which will need to be applied to a new 3.6 install.
- The 3.6 masterfiles can be found in a clean installation of CFEngine (hub package on Enterprise), under /var/cfengine/masterfiles. Apply any customizations against a copy of the 3.6 masterfiles in a well-known location, e.g.
/root/3.6/masterfiles
. - Use
cf-promises
to verify that the policy runs with 3.6, by runningcf-promises /root/3.6/masterfiles/promises.cf
andcf-promises /root/3.6/masterfiles/update.cf
. - Use
cf-promises
to verify that the policy runs with you previous version of CFEngine (e.g. 3.5), by running the same commands as above on a node with that CFEngine version. - The merged masterfiles should now be based on the 3.6 framework, include your policies and work on both the version you are upgrading from and with 3.6.
Set
trustkeysfrom
to trust all hosts, or at least the Policy Server, in the merged masterfiles policy, e.g. /root/3.6/masterfiles/update/update_policy.cf`.CFEngine 3.6 uses LMDB for local databases, whereas older versions of CFEngine typically use TokyoCabinet or QDBM. The classic networking protocol uses the
lastseen
database to verify that the mapping between a peer's IP address and the corresponding hostkey is not changed. Since the 3.6 installation will not have any mappings in the lastseen database, hosts won't trust the IP address of the policy server without that setting.Optional, Enterprise only: Export the data from your existing Enterprise MongoDB.
- Download the
cfmigrate
binary. - This binary will export user/role settings as well as long-living file-changes data from MongoDB.
- No other data will be exported, as it would either way expire after one week. If you need continued access to 3.5 compliance data, keep the 3.5 MongoDB available.
- The
cfmigrate
tool usesmongoexport
, which is available from mongodb.org (it is included in the hub package of CFEngine 3.5).
- Download the
On your existing Policy Server, stop the CFEngine services.
service cfengine3 stop
- Verify that the output of
ps -e | grep cf
is empty.
Clients will continue to execute the policy that they have.
Make a backup of the Policy Server, a full backup of
/var/cfengine
(or yourWORKDIR
equivalent) is recommended.cp -r /var/cfengine/ppkeys/ /root/3.5/ppkeys
tar cvzf /root/3.5/cfengine.tar.gz /var/cfengine
Save the list of hosts currently connecting to the Policy Server.
cf-key -s > /root/3.5/hosts
Perform the upgrade of the Policy Server (3.5 to 3.6)
- Ensure the CFEngine services are still stopped (only on the Policy Server).
- Verify that the output of
ps -e | grep cf
is empty.
- Verify that the output of
Uninstall the previous CFEngine package to start fresh (you may need to adjust the package name based on the CFEngine edition).
- Red Hat based distributions:
rpm -e cfengine-nova-hub rm -rf /var/cfengine
- Debian based distributions:
dpkg -p cfengine-nova-hub rm -rf /var/cfengine
Install the new CFEngine Policy Server package (you may need to adjust the package name based on CFEngine edition, version and distribution).
- Red Hat based distributions:
rpm -i cfengine-nova-hub-3.6.1.x86_64.rpm
- Debian based distributions:
dpkg --install cfengine-nova-hub_3.6.1-1_amd64.deb
Restore ppkeys and any Enterprise license file (default location is /var/cfengine/masterfiles/license.dat) from backup.
cp /root/3.5/ppkeys/* /var/cfengine/ppkeys/
Copy the merged masterfiles from the perparation you did above.
rm -rf /var/cfengine/masterfiles/* cp /root/3.6/masterfiles/* /var/cfengine/masterfiles/
Bootstrap the Policy Server to itself.
/var/cfengine/bin/cf-agent -B <POLICY-SERVER-IP>
Any error messages regarding processes can be corrected by running
cf-agent -f update.cf -IK
Optional: Import data previously exported from MongoDB using the
cfmigrate
binary.- Verify that users can log into Mission Portal.
Take the Policy Server online.
- Verify with
cf-key -s
that connections from all clients have been established within 5-10 minutes. - Select some clients to confirm that they have received the new policy and are running it without error.
- Verify with
Prepare Client upgrade (all versions)
- Make client packages available on the Policy Server in
/var/cfengine/master_software_updates
, under the appropriate directories for the OS distributions you use. - Turn on the auto-upgrade policy by setting class
trigger_upgrade
inupdate.cf
for a small set of clients, i.e. change!any
to an appropriate class like an IP network (e.g.ipv4_10_10_1|ipv4_10_10_2
). Verify that the selected hosts are upgrading successfully.
As an Enterprise user, confirm that the hosts start appearing in Mission Portal after 5-10 minutes. Otherwise, log manually into a set of hosts to confirm the successful upgrade.
Complete Client upgrade (all versions)
- Widen the group of hosts on which the
trigger_upgrade
class is set. - Continue to verify from
cf-key -s
or in the Enterprise Mission Portal that hosts are upgraded correctly and start reporting in. - Verify that the list of hosts you captured before the upgrade, e.g. in
/root/3.5/hosts
correspond to what you see is now reporting in.
Finalize (3.5 to 3.6)
- Reset the
trustkeysfrom
configuration to the previous value, typically an empty list. - Optional: Switch to the new 3.6 networking protocol to benefit from TLS encryption and
improved control attributes in
access
promises. Optional: Recursively purge
*.tcdb
and*.tcdb.lock
files from/var/cfengine
See
protocol_version
in Components and Common Control andallowlegacyconnects
inbody server control
.
Version Control
CFEngine is policy is stored in /var/cfengine/masterfiles
on the policy
server. It is common that this directory is backed by a version control system
(VCS), such as git or subversion. In this document we will focus on git, but
CFEngine is VCS agnostic.
Please note that the following applies to CFEngine Community or Enterprise, but in Enterprise there are built-in facilities that can make the following unnecessary. Please see Version Control and Configuration Policy for details.
Repository synchronization
When /var/cfengine/masterfiles
is backed by VCS, it may be useful to have an
agent policy that periodically checks the VCS server for the latest version
fetches any updates. Again, note that CFEngine Enterprise has this built-in.
After installing CFEngine on the policy server and before bootstrapping the agent to itself, we create a git clone of our masterfiles.
$ git clone git@github.com:upstream/masterfiles.git /var/cfengine/masterfiles/
The following policy uses git pull
with the --ff-only flag to avoid
potentially bad merges. This assumes that no development takes place in
/var/cfengine/masterfiles
itself.
bundle agent vcs_update
{
commands:
"/usr/bin/git"
args => "pull --ff-only origin master",
contain => masterfiles_contain;
}
body contain masterfiles_contain
{
chdir => "/var/cfengine/masterfiles";
}
Commit hooks
Commit hooks are scripts that are run when a repository is updated. We can use
a hook to notify a policy developer if an update causes a syntax error. While
the agent on the policy server should not copy from
/var/cfengine/masterfiles
to /var/cfengine/inputs
if the new policy does
not pass validation, it can nevertheless be helpful to employ VCS commit
hooks. A hook needs to be installed on the VCS server. Git and subversion
store their hooks on the server, under directories .git/hooks
and hooks
,
respectively.
Example git update hook
We can use a git update hook to prevent a change from being made unless it
passes syntax checking. The idea is to check out the revision in a temporary
directory and run cf-promises
on it. Here is an example hook.
#!/bin/sh
# --- Command line
REF_NAME="$1"
OLD_REV="$2"
NEW_REV="$3"
GIT=/usr/bin/git
TAR=/bin/tar
CF_PROMISES=/home/a10021/Source/core/cf-promises/cf-promises
TMP_CHECKOUT_DIR=/tmp/cfengine-post-commit-syntax-check/
MAIN_POLICY_FILE=promises.cf
echo "Creating temporary checkout directory at ${TMP_CHECKOUT_DIR}"
mkdir -p ${TMP_CHECKOUT_DIR}
echo "Clearing potential data in temporary checkout directory"
rm -rf ${TMP_CHECKOUT_DIR}/*
rm -rf ${TMP_CHECKOUT_DIR}/.svn
echo "Checking out revision ${REV} from ${REPOS} to file://${TMP_CHECKOUT_DIR}"
${GIT} archive ${NEW_REV} | tar -x -C ${TMP_CHECKOUT_DIR}
if [ $? -ne 0 ]; then
echo "Error checking out repository to temporary folder during post-commit syntax checking!" >&2
return 1
fi
echo "Running cf-promises -cf on ${TMP_CHECKOUT_DIR}/${MAIN_POLICY_FILE}"
${CF_PROMISES} -cf ${TMP_CHECKOUT_DIR}/${MAIN_POLICY_FILE}
if [ $? -ne 0 ]; then
echo "There were policy errors in pushed revision ${REV}" >&2
return 1
else
echo "Policy check completed successfully!"
return 0
fi
Example subversion post-commit hook
For subversion, the principle is essentially the same. Note that for a post-commit hook the check is run after update, so the repository may be left with a syntax error, but the committer is notified.
#!/bin/sh
REPOS="$1"
REV="$2"
SVN=/usr/bin/svn
CF_PROMISES=/home/a10021/Source/core/cf-promises/cf-promises
TMP_CHECKOUT_DIR=/tmp/cfengine-post-commit-syntax-check/
MAIN_POLICY_FILE=trunk/promises.cf
echo "Creating temporary checkout directory at ${TMP_CHECKOUT_DIR}"
mkdir -p ${TMP_CHECKOUT_DIR}
echo "Clearing potential data in temporary checkout directory"
rm -rf ${TMP_CHECKOUT_DIR}/*
rm -rf ${TMP_CHECKOUT_DIR}/.svn
echo "Checking out revision ${REV} from ${REPOS} to file://${TMP_CHECKOUT_DIR}"
${SVN} co -r ${REV} file://${REPOS} ${TMP_CHECKOUT_DIR}
if [ $? -ne 0 ]; then
echo "Error checking out repository to temporary folder during post-commit syntax checking!" >&2
return 1
fi
echo "Running cf-promises -cf on ${TMP_CHECKOUT_DIR}/${MAIN_POLICY_FILE}"
${CF_PROMISES} -cf ${TMP_CHECKOUT_DIR}/${MAIN_POLICY_FILE}
if [ $? -ne 0 ]; then
echo "There were policy errors in committed revision ${REV}" >&2
return 1
else
echo "Policy check completed successfully!"
return 0
fi
Writing and Serving Policy
- About Policies and Promises
- Policy Workflow
- How Promises Work
- Best Practices
- Layers of Abstraction in Policy
- Promises Available in CFEngine
- Authoring Policy Tools & Workflow
About Policies and Promises
Central to CFEngine's effectiveness in system administration is the concept of a "promise," which defines the intent and expectation of how some part of an overall system should behave.
CFEngine emphasizes the promises a client makes to the overall CFEngine network. Combining promises with patterns to describe where and when promises should apply is what CFEngine is all about.
This document describes in brief what a promise is and what a promise does. There are other resources for finding out additional details about "promises" in the See Also section at the end of this document.
What Are Promises
A promise is the documentation or definition of an intention to act or behave in some manner. They are the rules which CFEngine clients are responsible for implementing.
The Value of a Promise
When you make a promise it is an effort to improve trust, which is an economic time-saver. If you have trust then there is less need to verify, which in turn saves time and money.
When individual components are empowered with clear guidance, independent decision making power, and the trust that they will fulfil their duties, then systems that are complex and scalable, yet still manageable, become possible.
Anatomy of a Promise
bundle agent hello_world
{
reports:
any::
"Hello World!"
comment => "This is a simple promise saying hello to the world.";
}
How Promises Work
Everything in CFEngine can be thought of as a promise to be kept by different resources in the system. In a system that delivers a web site using Apache, an important promise may be to make sure that the httpd
or apache
package is installed, running, and accessible on port 80.
Summary for Writing, Deploying and Using Promises
Writing, deploying, and using CFEngine promises
will generally follow these simple steps:
- Using a text editor, create a new file (e.g.
hello_world.cf
). - Create a bundle and promise in the file (see "Hello World" Policy Example).
- Save the file on the policy server somewhere under
/var/cfengine/masterfiles
(can be under a sub-directory). - Let CFEngine know about the
promise
on thepolicy server
, generally in the file/var/cfengine/masterfiles/promises.cf
, or a file elsewhere but referred to inpromises.cf
. * Optional: it is also possible to call a bundle manually, usingcf-agent
. - Verify the
policy file
was deployed and successfully run.
See Tutorial for Running Examples for a more detailed step by step tutorial.
Policy Workflow
CFEngine does not make absolute choices for you, like other tools. Almost everything about its behavior is a matter of policy and can be changed.
In order to keep operations as simple as possible, CFEngine maintains a
private working directory on each machine, referred to in documentation as WORKDIR
and in policy by
the variable $(sys.workdir) By default, this is located at
/var/cfengine
or C:\var\CFEngine
. It contains everything CFEngine needs to
run.
The figure below shows how decisions flow through the parts of a system.
It makes sense to have a single point of coordination. Decisions are therefore usually made in a single location (the Policy Definition Point). The history of decisions and changes can be tracked by a version control system of your choice (e.g. git, Subversion, CVS etc.).
Decisions are made by editing CFEngine's policy file
promises.cf
(or one of its included sub-files). This process is carried out off-line.Once decisions have been formalized and coded, this new policy is copied to a decision distribution point, $(sys.masterdir) which defaults to
/var/cfengine/masterfiles
on all policy distribution servers.Every client machine contacts the policy server and downloads these updates. The policy server can be replicated if the number of clients is very large, but we shall assume here that there is only one policy server.
Once a client machine has a copy of the policy, it extracts only those promise proposals that are relevant to it, and implements any changes without human assistance. This is how CFEngine manages change.
CFEngine tries to minimize dependencies by decoupling processes. By following this pull-based architecture, CFEngine will tolerate network outages and will recover from deployment errors easily. By placing the burden of responsibility for decision at the top, and for implementation at the bottom, we avoid needless fragility and keep two independent quality assurance processes apart.
Best Practices
Policy Style Guide This covers punctuation, whitespace, and other styles to remember when writing policy.
Bundles Best Practices Refer to this page as you decide when to make a bundle and when to use classes and/or variables in them.
Testing Policies This page describes how to locally test CFEngine and play with configuration files.
See Also
Layers of Abstraction in Policy
CFEngine offers a number of layers of abstraction. The most fundamental atom in CFEngine is the promise. Promises can be made about many system issues, and you described in what context promises are to be kept.
CFEngine is designed to handle high level simplicity (without sacrificing low level capability) by working with configuration patterns. After all, configuration is all about promising consistent patterns in the resources of the system. Lists, for instance, are a particularly common kind of pattern: for each of the following... make a similar promise. There are several ways to organize patterns, using containers, lists and associative arrays.
Menu level
At this high level, a user selects
from a set of pre-defined services
(or
bundles in CFEngine parlance). The selection is not made by every host, rather
one places hosts into roles that will keep certain promises.
bundle agent service_catalogue # menu
{
methods:
any:: # selected by everyone
"everyone" usebundle => time_management,
comment => "Ensure clocks are synchronized";
"everyone" usebundle => garbage_collection,
comment => "Clear junk and rotate logs";
mailservers:: # selected by hosts in class
"mail server" -> { "goal_3", "goal_1", "goal_2" }
usebundle => app_mail_postfix,
comment => "The mail delivery agent";
"mail server" -> goal_3,
usebundle => app_mail_imap,
comment => "The mail reading service";
"mail server" -> goal_3,
usebundle => app_mail_mailman,
comment => "The mailing list handler";
}
Bundle level
At this level, users can switch on and off predefined features, or re-use standard methods, e.g. for editing files:
body common control
{
bundlesequence => {
webserver("on"),
dns("on"),
security_set("on"),
ftp("off")
};
}
The set of bundles that can be selected from is extensible by the user.
Promise level
This is the most detailed level of configuration, and gives full convergent promise behavior to the user. At this promise level, you can specify every detail of promise-keeping behavior, and combine promises together, reusing bundles and methods from standard libraries, or creating your own.
bundle agent addpasswd
{
vars:
# want to set these values by the names of their array keys
"pwd[mark]" string => "mark:x:1000:100:Mark B:/home/mark:/bin/bash";
"pwd[fred]" string => "fred:x:1001:100:Right Said:/home/fred:/bin/bash";
"pwd[jane]" string => "jane:x:1002:100:Jane Doe:/home/jane:/bin/bash";
files:
"/etc/passwd" # Use standard library functions
create => "true",
comment => "Ensure listed users are present",
perms => mog("644","root","root"),
edit_line => append_users_starting("addpasswd.pwd");
}
Promises Available in CFEngine
meta - information about promise bundles
Meta-data promises have no internal function. They are intended to be used to represent arbitrary information about promise bundles. Formally, meta promises are implemented as variables, and the values map to a variable context called bundlename_meta. The values can be used as variables and will appear in CFEngine Enterprise variable reports.
See meta
.
vars - a variable, representing a value
Variables in CFEngine are defined as promises that an identifier of a certain type represents a particular value. Variables can be scalars or lists of types string, int, real or data.
The allowed characters in variable names are alphanumeric (both upper and lower case) and undercore. Associative arrays using the string type and square brackets [] to enclose an arbitrary key are being deprecated in favor of the data variable type.
See vars
.
defaults - a default value for bundle parameters
Defaults promises are related to variables. If a variable or parameter in a promise bundle is undefined, or its value is defined to be invalid, a default value can be promised instead.
CFEngine does not use Perl semantics: i.e. undefined variables do not map to the empty string, they remain as variables for possible future expansion. Some variables might be defined but still contain unresolved variables. To handle this you will need to match the $(abc)
form of the variables.
See defaults
.
classes - a class, representing a state of the system
Classes promises may be made in any bundle. Classes that are set in common bundles are global in scope, while classes in all other bundles are local.
Note: The term class and context are sometimes used interchangeably.
See classes
.
users - add or remove users
User promises are promises made about local users on a host. They express which users should be present on a system, and which attributes and group memberships the users should have.
Every user promise has at least one attribute, policy, which describes whether or not the user should be present on the system. Other attributes are optional; they allow you to specify UID, home directory, login shell, group membership, description, and password.
A bundle can be associated with a user promise, such as when a user is created in order to do housekeeping tasks in his/her home directory, like putting default configuration files in place, installing encryption keys, and storing a login picture.
History: Introduced in CFEngine 3.6.0
See users
.
files - configure a file
Files promises are an umbrella for attributes of files. Operations fall basically into three categories: create, delete and edit.
See files
.
packages - install a package
CFEngine supports a generic approach to integration with native operating support for packaging. Package promises allow CFEngine to make promises regarding the state of software packages conditionally, given the assumption that a native package manager will perform the actual manipulations. Since no agent can make unconditional promises about another, this is the best that can be achieved.
See packages
.
guest_environments
Guest environment promises describe enclosed computing environments that can host physical and virtual machines, Solaris zones, grids, clouds or other enclosures, including embedded systems. CFEngine will support the convergent maintenance of such inner environments in a fixed location, with interfaces to an external environment.
CFEngine currently seeks to add convergence properties to existing interfaces for automatic self-healing of guest environments. The current implementation integrates with libvirt, supporting host virtualization for Xen, KVM, VMWare, etc. Thus CFEngine, running on a virtual host, can maintain the state and deployment of virtual guest machines defined within the libvirt framework. Guest environment promises are not meant to manage what goes on within the virtual guests. For that purpose you should run CFEngine directly on the virtual machine, as if it were any other machine.
See guest_environments
.
methods - take on a whole bundle of other promises
Methods are compound promises that refer to whole bundles of promises. Methods may be parameterized. Methods promises are written in a form that is ready for future development. The promiser object is an abstract identifier that refers to a collection (or pattern) of lower level objects that are affected by the promise-bundle. Since the use of these identifiers is for the future, you can simply use any string here for the time being.
See methods
.
processes - start or terminate processes
Process promises refer to items in the system process table, i.e., a command in some state of execution (with a Process Control Block). Promiser objects are patterns that are unanchored, meaning that they match line fragments in the system process table.
See processes
.
services - start or stop services
A service is a set of zero or more processes. It can be zero if the service is not currently running. Services run in the background, and do not require user intervention.
Service promises may be viewed as an abstraction of process and commands promises. An important distinguisher is however that a single service may consist of multiple processes. Additionally, services are registered in the operating system in some way, and get a unique name. Unlike processes and commands promises, this makes it possible to use the same name both when it is running and not.
Some operating systems are bundled with a lot of unused services that are running as default. At the same time, faulty or inherently insecure services are often the cause of security issues. With CFEngine, one can create promises stating the services that should be stopped and disabled.
The operating system may start a service at boot time, or it can be started by CFEngine. Either way, CFEngine will ensure that the service maintains the correct state (started, stopped, or disabled). On some operating systems, CFEngine also allows services to be started on demand, when they are needed. This is implemented though the inetd or xinetd daemon on Unix. Windows does not support this.
CFEngine also allows for the concept of dependencies between services, and can automatically start or stop these, if desired. Parameters can be passed to services that are started by CFEngine.
See services
.
commands - execute a command
Commands and processes are separated cleanly. Restarting of processes must be coded as a separate command. This stricter type separation allows for more careful conflict analysis to be carried out.
See commands
.
storage - verify attached storage
Storage promises refer to disks and filesystem properties.
See storage
.
databases - configure a database
CFEngine can interact with commonly used database servers to keep promises about the structure and content of data within them.
There are two main cases of database management to address: small embedded databases and large centralized databases.
Databases are often centralized entities that have a single point of management. While large monolithic database can be more easily managed with other tools, CFEngine can still monitor changes and discrepancies. In addition, CFEngine can also manage smaller embedded databases that are distributed in nature, whether they are SQL, registry or future types.
For example, creating 100 new databases for test purposes is a task for CFEngine; but adding a new item to an important production database is not a recommended task for CFEngine.
See databases
.
access - grant or deny access to file objects
Access promises are conditional promises made by resources living on the server.
The promiser is the name of the resource affected and is interpreted to be a path, unless a different resource_type is specified. Access is then granted to hosts listed in admit_ips, admit_keys and admit_hostnames, or denied using the counterparts deny_ips, deny_keys and deny_hostnames.
You layer the access policy by denying all access and then allowing it only to selected clients, then denying to an even more restricted set.
See access
.
roles - allow certain users to activate certain classes
Roles promises are server-side decisions about which users are allowed to define soft-classes on the server's system during remote invocation of cf-agent. This implements a form of Role Based Access Control (RBAC) for pre-assigned class-promise bindings. The user names cited must be attached to trusted public keys in order to be accepted. The regular expression is anchored, meaning it must match the entire name.
See roles
.
measurements - measure or sample data from the system
This is an Enterprise-only feature.
By default,CFEngine's monitoring component cf-monitord records performance data about the system. These include process counts, service traffic, load average and CPU utilization and temperature when available.
CFEngine Enterprise extends this in two ways. First it adds a three year trend summary based any 'shift'-averages. Second, it adds customizable measurements promises to monitor or log very specific user data through a generic interface. The end-result is to either generate a periodic time series, like the above mentioned values, or to log the results to custom-defined reports.
Promises of type measurement are written just like all other promises within a bundle destined for the agent concerned, in this case monitor. However, it is not necessary to add them to the bundlesequence, because cf-monitord executes all bundles of type monitor.
See measurements
.
reports - report a message
Reports promises simply print messages. Outputting a message without qualification can be a dangerous operation. In a large installation it could unleash an avalanche of messaging.
See reports
.
Authoring Policy Tools & Workflow
There are several ways to approach authoring promises and ensuring they are copied into and then deployed properly from the masterfiles
directory:
- Create or modify files directly in the
masterfiles
directory. - Copy new or modified files into the
masterfiles
directory (e.g. local file copy usingcp
,scp
overssh
). - Utilize a version control system (e.g. Git) to push/pull changes or add new files to the
masterfiles
directory. - Utilize CFEngine Enterprise's integrated Git respository. The CFEngine Enterprise Guide contains more information.
Authoring on a Workstation and Pushing to the Hub Using Git + GitHub
General Summary
- The "masterfiles" directory contains the promises and other related files (this is true in all situations).
- Replace the out of the box setup with an initialized
git
repository and remote to a clone hosted on GitHub. - Add a promise to
masterfiles
that tells CFEngine to check thatgit
repository for changes, and if there are any to merge them intomasterfiles
. - When an author wants to create a new promise, or modify an existing one, they clone the same repository on GitHub so that they have a local copy on their own computer.
- The author will make their edits or additions in their local version of the
masterfiles
repository. - After the author is done making their changes commit them using
git commit
. - After the changes are committed they are then pushed back to the remote repository on GitHub.
- As described in step3, CFEngine will pull any new changes that were pushed to GitHub (sometime within a five minute time interval).
- Those changes will first exist in
masterfiles
, and then afterwards will be deployed to CFEngine hosts that are bootstrapped to the hub.
Create a Repository on GitHub for Masterfiles
There are two methods possible with GitHub: one is to use the web interface at GitHub.com; the second is to use the GitHub application.
Method One: Create Masterfiles Repository Using GitHub Web Interface
1a. In the GitHub web interface, click on the New repository
button.
1b. Or from the +
drop down menu on the top right hand side of the screen select New repository
.
2. Fill in a value in the Repository name
text entry (e.g. cfengine-masterfiles).
3. Select private
for the type of privacy desired (public
is also possible, but is not recommended in most situations).
4. Optionally, check the Initialize this repository with a README
box. (not required):""
Method Two: Create Masterfiles Repository Using the GitHub Application
- Open the GitHub app and click on the "+ Create" sign to create a new repository.
- Fill in a value in the
Repository name
text entry (e.g. cfengine-masterfiles). - Select
private
for the type of privacy desired (public
is also possible, but is not recommended in most situations). - Select one of your "Accounts" where you want the new repository to be created.
- Click on the "Create" button at the bottom of the screen. A new repository will be created in your local GitHub folder.
Initialize Git Repository in Masterfiles on the Hub
> cd /var/cfengine/masterfiles
> git init
> git commit -m "First commit"
> git remote add origin https://github.com/GitUserName/cfengine-masterfiles.git
> git push -u origin master
Using the above steps on a private repository will fail with a 403 error. There are different approaches to deal with this:
A) Generate a key pair and add it to GitHub
- As root, type
ssh-keygen -t rsa
. - Hit enter when prompted to
Enter file in which to save the key (/root/.ssh/id_rsa):
. - Hit enter again when prompted to
Enter passphrase (empty for no passphrase):
. - Type
ssh-agent bash
and then the enter key. - Type
ssh-add /root/.ssh/id_rsa
. - Type
exit
to leavessh-agent bash
. - To test, type
ssh -T git@github.com
. - Open the generated key file (e.g.
vi /root/.ssh/id_rsa.pub
). - Copy the contents of the file to the clipboard (e.g. Ctrl+Shift+C).
- In the GitHub web interface, click the user account settings button (the icon with the two tools in the top right hand corner).
- On the next screen, on the left hand side, click
SSH keys
. - Click
Add SSH key
on the next screen. - Provide a
Title
for the label (e.g. CFEngine). - Paste the key contents from the clipboard into the
Key
textarea. - Click
Add key
. - If prompted to do so, provide your GitHub password, and then click the
Confirm
button.
B) Or, change the remote url to https://GitUserName@password:github.com/GitUserName/cfengine-masterfiles.git
. This is not safe in a production environment and should only be used for basic testing purposes (if at all).
Create a Remote in Masterfiles on the Hub to Masterfiles on GitHub
- Change back to the
masterfiles
directory, if not already there:> cd /var/cfengine/masterfiles
- Create the remote using the following pattern:
> git remote add upstream ssh://git@github.com/GitUserName/cfengine-masterfiles.git
.
- Verify the remote was registered properly by typing
git remote -v
and pressing enter.- You will see the remote definition in a list alongside any other previously defined remote enteries.
Add a Promise that Pulls Changes to Masterfiles on the Hub from Masterfiles on GitHub
- Create a new file in
/var/cfengine/masterfiles with a unique filename
(e.g.vcs_update.cf
) - Add the following text to the
vcs_update.cf
file:
bundle agent vcs_update
{
commands:
"/usr/bin/git"
args => "pull --ff-only upstream master",
contain => masterfiles_contain;
}
body contain masterfiles_contain
{
chdir => "/var/cfengine/masterfiles";
}
- Save the file.
- Add bundle and file information to
/var/cfengine/masterfiles/promises.cf
. Example (where...
represents existing text in the file, omitted for clarity:
body common control
{
bundlesequence => {
...
vcs_update,
};
inputs => {
...
"vcs_update.cf",
};
- Save the file.
<!--- End include:
/home/jenkins/workspace/build-documentation-3.6/label/DOCUMENTATION_x86_64_linux_debian_6/documentation/guide/writing-and-serving-policy/authoring-policy-tools-and-workflow.markdown
-->
Policy Style Guide
Style is a very personal choice and the contents of this guide should only be considered suggestions. We invite you to contribute to the growth of this guide.
Style Summary
- one indent = 2 spaces
- avoid letting line length surpass 80 characters.
- vertically align opening and closing curly braces unless on same line
- promise type = 1 indent
- context class expression = 2 indents
- promiser = 3 indents
- promise attributes = (we suggest 3 or 4 indents)
Whitespace and Line Length
Spaces are preferred to tab characters. Lines should not have trailing whitespace. Generally line length should not surpass 80 characters.
Curly brace alignment
Generally if opening and closing braces are not on a single line they should be aligned vertically.
Example:
bundle agent example
{
vars:
"people" slist => {
"Obi-Wan Kenobi",
"Luke Skywalker",
"Chewbacca",
"Yoda",
"Darth Vader",
};
"cuddly" slist => { "Chewbacca", "Yoda" };
}
Promise types
Promise types should have 1 indent and each promise type after the first listed should have a blank line before the next promise type.
This example illustrates the blank line before the "classes" type.
bundle agent example
{
vars:
"policyhost" string => "MyPolicyServerHostname";
classes:
"EL5" or => { "centos_5", "redhat_5" };
"EL6" or => { "centos_6", "redhat_6" };
}
Context class expressions
Context class expressions should have 2 indents and each context class expression after the first listed within a given promise type should have a blank line preceding it.
This example illustrates the blank line before the second context class expression (solaris) in the files type promise section:
bundle agent example
{
files:
any::
"/var/cfengine/inputs/"
copy_from => update_policy( "/var/cfengine/masterfiles","$(policyhost)" ),
classes => policy_updated( "policy_updated" ),
depth_search => recurse("inf");
solaris::
"/var/cfengine/inputs"
copy_from => update_policy( "/var/cfengine/masterfiles", "$(policyhost" ),
classes => policy_updated( "policy_updated" );
}
Policy Comments
In-line policy comments are useful for debugging and explaining why something is done a specific way. We encourage you to document your policy thoroughly.
Comments about general body and bundle behavior and parameters should be placed after the body or bundle definition, before the opening curly brace and should not be indented. Comments about specific promise behavior should be placed before the promise at the same indention level as the promiser or on the same line after the attribute.
bundle agent example(param1)
# This is an example bundle to illustrate comments
# param1 - string -
{
vars:
"copy_of_param1" string => "$(param1)";
"jedi" slist => {
"Obi-Wan Kenobi",
"Luke Skywalker",
"Yoda",
"Darth Vader", # He used to be a Jedi, and since he
# tossed the emperor into the Death
# Star's reactor shaft we are including
# him.
};
classes:
# Most of the time we don't need differentiation of redhat and centos
"EL5" or => { "centos_5", "redhat_5" };
"EL6" or => { "centos_6", "redhat_6" };
}
Policy Reports
It is common and useful to include reports in policy to get detailed information about what is going on. During a normal agent run the goal is to have 0 output so reports should always be guarded with a class. Carefully consider when your policy should generate report output. For policy degbugging type information (value of variables, classes that were set or not) the following style is reccomended:
bundle agent example
{
reports:
DEBUG|DEBUG_example::
"DEBUG $(this.bundle): Desired Report Output";
}
Following this style keeps policy debug reports from spamming logs. It avoids
polluting the inform_mode
and verbose_mode
output, and it allows you to get
debug output for ALL policy or just a select bundle which is incredibly useful
when debugging a large policy set.
Promise Handles
Promise handles uniquely identify a promise within a policy. We suggest a simple naming
scheme of bundle_name_promise_type_class_restriction_promiser
to keep handles unique and
easily identifiable. Often it may be easier to omit the handle.
bundle agent example
{
commands:
dev::
"/usr/bin/git"
args => "pull",
contain => in_dir("/var/srv/myrepo"),
ifvarclass => "redhat",
handle => "example_commands_dev_redhat_git_pull";
}
Hashrockets (=>)
You may align hash rockets within a promise body scope and for grouped single line promises.
Example:
bundle agent example
{
files:
any::
"/var/cfengine/inputs/"
copy_from => update_policy( "/var/cfengine/masterfiles","$(policyhost)" ),
classes => policy_updated( "policy_updated" ),
depth_search => recurse("inf");
"/var/cfengine/modules"
copy_from => update_policy( "/var/cfengine/modules", "$(policyhost" ),
classes => policy_updated( "modules_updated" );
classes:
"EL5" or => { "centos_5", "redhat_5" };
"EL6" or => { "centos_6", "redhat_6" };
}
You may also simply leave them as they are:
bundle agent example
{
files:
any::
"/var/cfengine/inputs/"
copy_from => update_policy( "/var/cfengine/masterfiles","$(policyhost)" ),
classes => policy_updated( "policy_updated" ),
depth_search => recurse("inf");
"/var/cfengine/modules"
copy_from => update_policy( "/var/cfengine/modules", "$(policyhost" ),
classes => policy_updated( "modules_updated" );
classes:
"EL5" or => { "centos_5", "redhat_5" };
"EL6" or => { "centos_6", "redhat_6" };
}
Which one do you prefer?
Naming Conventions
Internal variables & classes
Variables and classes that have no centralized reporting value are considered "internal". By convention internal variables and classes should be prefixed with an underscore "_".
Deprecating Bundles
As your policy library changes over time you may want to deprecate various bundles in favor of newer implimentations. To indicate that a bundle is deprecated we recommend the following style.
bundle agent old
{
meta:
"tags" slist => {
"deprecated=3.6.0",
"deprecation-reason=More feature rich implimentation",
"replaced-by=newbundle",
};
}
Automatic reindentation
You can run contrib/reindent.pl FILE1.cf FILE2.c FILE3.h
to reindent files,
if you don't want to set up Emacs. It will rewrite them with the new
indentation, using Emacs in batch mode.
Bundles Best Practices
The following contains practices to remember when creating bundles as you write policy.
How to choose and name bundles
Use the name of a bundle to represent a meaningful aspect of system administration, We recommend using a two- or three-part name that explains the context, general subject heading, and special instance. Names should be service-oriented in order to guide non-experts to understand what they are about.
For example:
- app_mail_postfix
- app_mail_mailman
- app_web_apache
- app_web_squid
- app_web_php
- app_db_mysql
- garbage_collection
- security_check_files
- security_check_processes
- system_name_resolution
- system_xinetd
- system_root_password
- system_processes
- system_files
- win_active_directory
- win_registry
- win_services
When to make a bundle
Put items into a single bundle if:
- They belong to the same conceptual aspect of system administration.
- They do not need to be switched on or off independently.
Put items into different bundles if:
- All of the promises in one bundle need to the checked before all of the promises in another bundle.
- You need to re-use the promises with different parameters.
In general, keep the number of bundles to a minimum. This is a knowledge-management issue. Clarity comes from differentiation, but only if the number of items is small.
When to use a paramaterized bundle or method
If you need to arrange for a managed convergent collection or sequence of promises that will occur for a list of (multiple) names or promisers, then use a bundle to simplify the code.
Write the promises (which may or may not be ordered) using a parameter for the different names, and then call the method passing the list of names as a parameter to reduce the amount of code.
bundle agent testbundle
{
vars:
"userlist" slist => { "mark", "jeang", "jonhenrik", "thomas", "eben" };
methods:
"any" usebundle => subtest("$(userlist)");
}
###########################################
bundle agent subtest(user)
{
commands:
"/bin/echo Fix $(user)";
files:
"/home/$(user)/."
create => "true";
reports:
linux::
"Finished doing stuff for $(user)";
}
When to use classes in common bundles
- When you need to use them in multiple bundles (because classes defined in common bundles have global scope).
When to use variables in common bundles
- For rationality, if the variable does not belong to any particular bundle, because it is
used elsewhere. (Qualified variable names such as
$(mybundle.myname)
are always globally accessible, so this is a cosmetic issue.)
When to use variables in local bundles
- If they are not needed outside the bundles.
- If they are used for iteration (without qualified scope).
- If they are tied to a specific aspect of system maintenance represented by the bundle, so
that accessing
$(bundle.var)
adds clarity.
The Policy Framework
The CFEngine policy framework is called the masterfiles because
the files live in /var/cfengine/masterfiles
on the policy server (on
the clients, and note the policy server is typically also a client,
they are cached in /var/cfengine/inputs
).
The following configuration files are part of the default CFEngine
installation in /var/cfengine/masterfiles
, and have special roles.
Setting up
First, review update.cf
and def.cf
. Most settings you need to change will live here.
update.cf
Synchronizing clients with the policy server happens here, in
update.cf
. Its main job is to copy all the files on the policy
server (usually the hub) under $(sys.masterdir)
(usually
/var/cfengine/masterfiles
) to the local host into $(sys.inputdir)
(usually /var/cfengine/inputs
).
This file should rarely if ever change. Should you ever change it (or
when you upgrade CFEngine), take special care to ensure the old and
the new CFEngine can parse and execute this file successfully. If not,
you risk losing control of your system (that is, if CFEngine cannot
successfully execute update.cf
, it has no mechanism for distributing
new policy files).
By default, the policy defined in update.cf is executed from two sets of
promise bodies. The "usual" one (defined in the bundlesequence
in
promises.cf
) and another in the backup/failsafe bundlesequence
(defined in
failsafe.cf
).
This is a standalone policy file. You can actually run it with
cf-agent -KI -f ./update.cf
but if you don't understand what that
command does, please hold off until you've gone through the CFEngine
documentation. The contents of update.cf
duplicate other things
under lib
sometimes, in order to be completely standalone.
To repeat, when update.cf
is broken, things go bonkers. CFEngine
will try to run a backup failsafe.cf
you can find in the C core
under libpromises/failsafe.cf
(that .cf
file is written into the C
code and can't be modified). If things get to that point, you probably
have to look at why corrupted policies made it into production.
As is typical for CFEngine, the policy and the configuration are
mixed. In update.cf
you'll find some very useful settings. Keep
referring to
update.cf
as you
read this. We are skipping the nonessential ones.
How it works
There are 4 stages in update.cf
. See the bundlesequence
: after
loading the configuration from update_def
, we take these steps in
order.
cfe_internal_dc_workflow
This step implements the auto-deployment of policies. See
Version Control and Configuration Policy
and cfengine_internal_masterfiles_update
below for details.
cfe_internal_update_policy
This step updates the policy files themselves. Basically it's a check
step that looks at $(sys.inputdir)/cf_promises_validated
and
compares it with the policy server's
$(sys.masterdir)/cf_promises_validated
. Then there's the actual
copy, which happens only if the cf_promises_validated
file was
updated in the check step.
Implementation (warning: advanced usage):
bundle agent cfe_internal_update_policy
{
vars:
!(cfengine_3_5||cfengine_3_4)::
"inputs_dir" string => translatepath("$(sys.inputdir)"),
comment => "Directory containing CFEngine policies",
handle => "cfe_internal_update_policy_vars_inputs_dir";
"dir_bin" string => translatepath("$(sys.bindir)"),
comment => "Directory containing CFEngine binaries",
handle => "cfe_internal_update_policy_vars_dir_bin";
(cfengine_3_5||cfengine_3_4)::
"inputs_dir" string => translatepath("$(sys.workdir)/inputs"),
comment => "Directory containing CFEngine policies",
handle => "cfe_internal_update_policy_vars_inputs_dir_backport";
"dir_bin" string => translatepath("$(sys.workdir)/bin"),
comment => "Directory containing CFEngine binaries",
handle => "cfe_internal_update_policy_vars_dir_bin_backport";
windows::
"master_location" string => "/var/cfengine/masterfiles", # NB! NOT $(sys.workdir) on Windows !
comment => "The master CFEngine policy directory on the policy host",
handle => "cfe_internal_update_policy_vars_master_location_windows";
"modules_dir" string => "/var/cfengine/modules", # NB! NOT $(sys.workdir) on Windows !
comment => "Directory containing CFEngine modules",
handle => "cfe_internal_update_policy_vars_modules_dir_windows";
"plugins_dir" string => "/var/cfengine/plugins", # NB! NOT $(sys.workdir) on Windows !
comment => "Directory containing CFEngine plugins",
handle => "cfe_internal_update_policy_vars_plugins_dir_windows";
!windows.!(cfengine_3_5||cfengine_3_4)::
"master_location" string => "$(sys.masterdir)",
comment => "The master CFEngine policy directory on the policy host",
handle => "cfe_internal_update_policy_vars_master_location";
!windows.(cfengine_3_5||cfengine_3_4)::
"master_location" string => "$(sys.workdir)/masterfiles",
comment => "The master CFEngine policy directory on the policy host",
handle => "cfe_internal_update_policy_vars_master_location_backport";
!windows::
"modules_dir" string => translatepath("$(sys.workdir)/modules"),
comment => "Directory containing CFEngine modules",
handle => "cfe_internal_update_policy_vars_modules_dir";
"plugins_dir" string => translatepath("$(sys.workdir)/plugins"),
comment => "Directory containing CFEngine plugins",
handle => "cfe_internal_update_policy_vars_plugins_dir";
any::
"file_check" string => translatepath("$(inputs_dir)/promises.cf"),
comment => "Path to a policy file",
handle => "cfe_internal_update_vars_file_check";
"ppkeys_file" string => translatepath("$(sys.workdir)/ppkeys/localhost.pub"),
comment => "Path to public key file",
handle => "cfe_internal_update_policy_vars_ppkeys_file";
"postgresdb_dir" string => "$(sys.workdir)/state/pg/data",
comment => "Directory where Postgres database files will be stored on hub -",
handle => "cfe_internal_update_policy_postgresdb_dir";
"postgresdb_log" string => "/var/log/postgresql.log",
comment => "File where Postgres database files will be logging -",
handle => "cfe_internal_update_policy_postgresdb_log_file";
"redis_conf_file" string => translatepath("$(sys.workdir)/config/redis.conf"),
comment => "Path to Redis configuration file",
handle => "cfe_internal_update_policy_redis_conf_file";
"agents" slist => {
"cf-agent",
"cf-execd",
"cf-key",
"cf-monitord",
"cf-promises",
"cf-runagent",
"cf-serverd",
},
comment => "List of CFEngine binaries",
handle => "cfe_internal_update_policy_vars_agents";
"optional_agents" slist => {
"cf-consumer",
"cf-hub",
},
comment => "List of optional CFEngine binaries",
handle => "cfe_internal_update_policy_vars_optional_agents";
#
classes:
"have_bindir_$(optional_agents)" expression => fileexists("$(dir_bin)/$(optional_agents)");
any::
"have_ppkeys" expression => fileexists("$(ppkeys_file)"),
comment => "Check for $(sys.workdir)/ppkeys/localhost.pub",
handle => "cfe_internal_update_policy_classes_have_ppkeys";
"local_files_ok" expression => fileexists("$(file_check)"),
comment => "Check for $(sys.masterdir)/promises.cf",
handle => "cfe_internal_update_classes_files_ok";
# create a global files_ok class
"cfe_internal_trigger" expression => "local_files_ok",
classes => u_if_else("files_ok", "files_ok");
#
commands:
!have_ppkeys::
"$(sys.cf_key)"
comment => "Generate cfengine encryption keys if necessary",
classes => u_kept_successful_command,
handle => "cfe_internal_update_policy_commands_generate_keys";
am_policy_hub.update_report::
"$(sys.cf_promises) -r"
contain => u_in_shell_and_silent,
comment => "Generate config knowledge format after update",
classes => u_kept_successful_command,
handle => "cfe_internal_update_policy_commands_run_cf_promises_r";
#
files:
!am_policy_hub:: # policy hub should not alter inputs/ uneccessary
"$(inputs_dir)/cf_promises_validated"
comment => "Check whether a validation stamp is available for a new policy update to reduce the distributed load",
handle => "cfe_internal_update_policy_check_valid_update",
copy_from => u_rcp("$(master_location)/cf_promises_validated", @(update_def.policy_servers)),
action => u_immediate,
classes => u_if_repaired("validated_updates_ready");
!am_policy_hub.!windows::
"$(modules_dir)"
comment => "Always update modules files on client side",
handle => "cfe_internal_update_policy_files_update_modules",
copy_from => u_rcp("$(modules_dir)", @(update_def.policy_servers)),
depth_search => u_recurse("inf"),
perms => u_m("755"),
action => u_immediate;
"$(plugins_dir)"
comment => "Always update plugins files on client side",
handle => "cfe_internal_update_policy_files_update_plugins",
copy_from => u_rcp("$(plugins_dir)", @(update_def.policy_servers)),
depth_search => u_recurse("inf"),
perms => u_m("755"),
action => u_immediate;
!am_policy_hub.windows::
"$(sys.workdir)\modules"
comment => "Always update modules files on client side (Windows)",
handle => "cfe_internal_update_policy_files_update_modules_windows",
copy_from => u_rcp("$(modules_dir)", @(update_def.policy_servers)),
depth_search => u_recurse("inf"),
perms => u_m("755"),
action => u_immediate;
"$(sys.workdir)\plugins"
comment => "Always update plugins files on client side (Windows)",
handle => "cfe_internal_update_policy_files_update_plugins_windows",
copy_from => u_rcp("$(plugins_dir)", @(update_def.policy_servers)),
depth_search => u_recurse("inf"),
perms => u_m("755"),
action => u_immediate;
am_policy_hub|validated_updates_ready:: # policy hub should always put masterfiles in inputs in order to check new policy
"$(inputs_dir)"
comment => "Copy policy updates from master source on policy server if a new validation was acquired",
handle => "cfe_internal_update_policy_files_inputs_dir",
copy_from => u_rcp("$(master_location)", @(update_def.policy_servers)),
depth_search => u_recurse("inf"),
file_select => u_input_files,
action => u_immediate,
classes => u_if_repaired("update_report");
!policy_server.enable_cfengine_enterprise_hub_ha::
"$(sys.workdir)/policy_server.dat"
comment => "Copy policy_server.dat file from server",
handle => "cfe_internal_update_ha_policy_server",
copy_from => u_rcp("$(sys.workdir)/state/master_hub.dat", @(update_def.policy_servers)),
action => u_immediate,
classes => u_if_repaired("replica_failover"); # not needed ?
!windows::
"$(dir_bin)/$(agents)"
comment => "Make sure cfengine binaries have right file permissions",
handle => "cfe_internal_update_policy_files_dir_bin_agents",
perms => u_m("755"),
action => u_immediate;
"$(dir_bin)/$(optional_agents)"
comment => "Make sure the optional cfengine binaries have right file permissions",
handle => "cfe_internal_update_policy_files_sys_workdir_bin_optional",
perms => u_m("755"),
action => u_immediate,
ifvarclass => canonify("have_bindir_$(optional_agents)");
"$(sys.workdir)/bin"
comment => "Make sure cfengine binaries have right file permissions",
handle => "cfe_internal_update_policy_files_sys_workdir_bin",
perms => u_m("755"),
depth_search => u_recurse_basedir("inf"),
ifvarclass => and(strcmp($(sys.workdir), "/var/cfengine")),
action => u_immediate;
"$(sys.workdir)/lib"
comment => "Make sure cfengine libraries have right file permissions",
handle => "cfe_internal_update_policy_files_sys_workdir_lib",
perms => u_shared_lib_perms,
depth_search => u_recurse_basedir("inf"),
action => u_immediate;
"/usr/local/sbin/$(agents)"
comment => "Create symlinks of CFE binaries in /usr/local/sbin",
handle => canonify("cfe_internal_update_policy_files_sbin_$(agents)"),
move_obstructions => "true",
link_from => u_ln_s("$(sys.workdir)/bin/$(agents)");
"/usr/local/sbin/$(agents).cfsaved"
comment => "Remove all .cfsaved file extension",
handle => canonify("cfe_internal_update_policy_files_remove_$(agent)_cfsaved"),
delete => u_tidy;
am_policy_hub::
"$(master_location)/."
comment => "Make sure masterfiles folder has right file permissions",
handle => "cfe_internal_update_policy_files_sys_workdir_masterfiles",
perms => u_m($(update_def.masterfiles_perms_mode)),
depth_search => u_recurse_basedir("inf"),
action => u_immediate;
}
body perms u_m(p)
{
mode => "$(p)";
}
body perms u_mo(p,o)
{
mode => "$(p)";
owners => {"$(o)"};
}
body perms u_shared_lib_perms
{
!hpux::
mode => "0644";
hpux::
mode => "0755"; # Mantis 1114, Redmine 1179
}
body file_select u_cf3_files
{
leaf_name => { "cf-.*" };
file_result => "leaf_name";
}
body file_select u_input_files
{
leaf_name => { @(update_def.input_name_patterns) };
file_result => "leaf_name";
}
body copy_from u_rcp(from,server)
{
source => "$(from)";
compare => "digest";
trustkey => "false";
!am_policy_hub::
servers => { "$(server)" };
cfengine_internal_encrypt_transfers::
encrypt => "true";
cfengine_internal_purge_policies::
purge => "true";
}
body copy_from u_cp(from)
{
source => "$(from)";
compare => "digest";
}
body copy_from u_cp_nobck(from)
{
source => "$(from)";
compare => "digest";
copy_backup => "false";
}
body action u_immediate
{
ifelapsed => "0";
}
body depth_search u_recurse(d)
{
depth => "$(d)";
exclude_dirs => { "\.svn", "\.git", "git-core" };
}
body depth_search u_recurse_basedir(d)
{
include_basedir => "true";
depth => "$(d)";
exclude_dirs => { "\.svn", "\.git", "git-core" };
}
body classes u_if_repaired(x)
{
promise_repaired => { "$(x)" };
}
body classes u_if_repaired_then_cancel(y)
{
cancel_repaired => { "$(y)" };
}
body classes u_if_else(yes,no)
{
promise_repaired => { "$(yes)" };
repair_failed => { "$(no)" };
repair_denied => { "$(no)" };
repair_timeout => { "$(no)" };
}
body contain u_in_shell
{
useshell => "true";
}
body contain u_in_shell_and_silent
{
useshell => "true";
no_output => "true";
}
body contain u_postgres
{
useshell => "useshell";
exec_owner => "cfpostgres";
chdir => "/tmp";
no_output => "true";
}
body action u_ifwin_bg
{
windows::
background => "true";
}
body service_method u_bootstart
{
service_autostart_policy => "boot_time";
}
body contain u_silent_in_dir(s)
{
chdir => "$(s)";
no_output => "true";
}
body link_from u_ln_s(x)
{
link_type => "symlink";
source => "$(x)";
when_no_source => "force";
}
body delete u_tidy
{
dirlinks => "delete";
rmdirs => "true";
}
cfe_internal_update_processes
This step manages the running processes, ensuring cf-execd
and
cf-serverd
and cf-monitord
are running and doing some other tasks.
cfe_internal_update_bins
This step does a self-update of CFEngine. See the Enterprise documentation for details; this functionality is unsupported in CFEngine Community.
update.cf configuration
input_name_patterns
Change this variable to add more file extensions to the list copied from the policy server. By default this is a pretty sparse list.
masterfiles_perms_mode
Usually you want to leave this at 0600
meaning the inputs will be
readable only by their owner.
cfengine_internal_masterfiles_update
Off by default.
Turn this on (set to any
) to auto-deploy policies on the policy
server, it has no effect on clients. See
Version Control and Configuration Policy
for details on how to use it.
This may result in DATA LOSS.
cfengine_internal_encrypt_transfers
Off by default.
Turn this on (set to any
) to encrypt your policy transfers.
Note it has a duplicate in def.cf
, see below. If they are not
synchronized, you will get unexpected behavior.
cfengine_internal_purge_policies
Off by default.
Turn this on (set to any
) to delete any files in your
$(sys.inputdir)
that are not in the policy server's masterfiles.
This may result in DATA LOSS.
Note it has a duplicate in def.cf
, see below. If they are not
synchronized, you will get unexpected behavior.
cfengine_internal_preserve_permissions
Off by default.
Turn this on (set to any
) to preserve the permissions of the policy
server's masterfiles when they are copied.
This may result in FUNCTIONALITY LOSS if your scripts lose their exec bits unexpectedly
Note it has a duplicate in def.cf
, see below. If they are not
synchronized, you will get unexpected behavior.
def.cf
After update.cf
is configured, you can configure the main def.cf
policy.
This file is included by the main promises.cf
and you can run that
with cf-agent -KI -f ./promises.cf
but as before, make sure you
understand what this command does before using it.
Keep referring to def.cf
as you read this.
Implementation (warning: advanced usage):
bundle common def
{
vars:
any::
# Begin change
# Your domain name, for use in access control
"domain" string => "$(sys.domain)", # this default may be inaccurate!
comment => "Define a global domain for all hosts",
handle => "common_def_vars_domain";
# Mail settings used by body executor control found in controls/cf_execd.cf
"mailto" string => "root@$(def.domain)";
"mailfrom" string => "root@$(sys.uqhost).$(def.domain)";
"smtpserver" string => "localhost";
# List here the IP masks that we grant access to on the server
"acl" slist => {
# Allow everything in my own domain.
# Note that this:
# 1. requires def.domain to be correctly set
# 2. will cause a DNS lookup for every access
".*$(def.domain)",
# Assume /16 LAN clients to start with
"$(sys.policy_hub)/16",
# "2001:700:700:3.*",
# "217.77.34.18",
# "217.77.34.19",
},
comment => "Define an acl for the machines to be granted accesses",
handle => "common_def_vars_acl";
# Out of the hosts in allowconnects, trust new keys only from the
# following ones. This is open by default for bootstrapping.
"trustkeysfrom" slist => {
# COMMENT THE NEXT LINE OUT AFTER ALL MACHINES HAVE BEEN BOOTSTRAPPED.
"0.0.0.0/0", # allow any IP
},
comment => "Define from which machines keys can be trusted";
# End change #
cfengine_3_4|cfengine_3_5::
"dir_masterfiles" string => translatepath("$(sys.workdir)/masterfiles"),
comment => "Define masterfiles path",
handle => "common_def_vars_dir_masterfiles_backport";
!cfengine_3_4.!cfengine_3_5::
"dir_masterfiles" string => translatepath("$(sys.masterdir)"),
comment => "Define masterfiles path",
handle => "common_def_vars_dir_masterfiles";
"dir_reports" string => translatepath("$(sys.workdir)/reports"),
comment => "Define reports path",
handle => "common_def_vars_dir_reports";
"dir_software" string => translatepath("$(sys.workdir)/master_software_updates"),
comment => "Define software path",
handle => "common_def_vars_dir_software";
"dir_bin" string => translatepath("$(sys.bindir)"),
comment => "Define binary path",
handle => "common_def_vars_dir_bin";
"dir_modules" string => translatepath("$(sys.workdir)/modules"),
comment => "Define modules path",
handle => "common_def_vars_dir_modules";
"dir_plugins" string => translatepath("$(sys.workdir)/plugins"),
comment => "Define plugins path",
handle => "common_def_vars_dir_plugins";
cfengine_3_4|cfengine_3_5::
"cf_apache_user" string => "apache",
comment => "User that CFEngine Enterprise webserver runs as",
handle => "common_def_vars_cf_apache_user";
"cf_apache_group" string => "apache",
comment => "Group that CFEngine Enterprise webserver runs as",
handle => "common_def_vars_cf_apache_group";
!cfengine_3_4.!cfengine_3_5::
"cf_apache_user" string => "cfapache",
comment => "User that CFEngine Enterprise webserver runs as",
handle => "common_def_vars_cf_cfapache_user";
"cf_apache_group" string => "cfapache",
comment => "Group that CFEngine Enterprise webserver runs as",
handle => "common_def_vars_cf_cfapache_group";
solaris::
"cf_runagent_shell"
string => "/usr/bin/sh",
comment => "Define path to shell used by cf-runagent",
handle => "common_def_vars_solaris_cf_runagent_shell";
!(windows|solaris)::
"cf_runagent_shell"
string => "/bin/sh",
comment => "Define path to shell used by cf-runagent",
handle => "common_def_vars_cf_runagent_shell";
any::
"base_log_files" slist =>
{
"$(sys.workdir)/cf3.$(sys.uqhost).runlog",
"$(sys.workdir)/promise_summary.log",
};
"enterprise_log_files" slist =>
{
"$(sys.workdir)/cf_notkept.log",
"$(sys.workdir)/cf_repair.log",
"$(sys.workdir)/state/cf_value.log",
"$(sys.workdir)/outputs/dc-scripts.log",
};
"hub_log_files" slist =>
{
"$(sys.workdir)/httpd/logs/access_log", # Mission Portal
"$(sys.workdir)/httpd/logs/error_log", # Mission Portal
};
"max_client_history_size" -> { "cf-hub", "CFEngine Enterprise" }
int => "50M",
comment => "The threshold of report diffs which will trigger purging of
diff files.";
enterprise.!am_policy_hub::
# CFEngine's own log files
"cfe_log_files" slist => { @(base_log_files), @(enterprise_log_files) };
enterprise.am_policy_hub::
# CFEngine's own log files
"cfe_log_files" slist => { @(base_log_files), @(enterprise_log_files), @(hub_log_files) };
# Number of days to keep scheduled reports before purging
"purge_scheduled_reports_older_than_days" -> { "Mission Portal", "CFEngine Enterprise" }
int => "30";
!enterprise::
# CFEngine's own log files
"cfe_log_files" slist => { @(base_log_files) };
any::
"cfe_log_dirs" slist =>
{
"$(sys.workdir)/outputs",
"$(sys.workdir)/reports",
};
# enable_cfengine_enterprise_hub_ha is defined below
# Disabled by default
enable_cfengine_enterprise_hub_ha::
"policy_servers" slist => {"$(sys.policy_hub)", @(ha_def.ips)};
!enable_cfengine_enterprise_hub_ha::
"policy_servers" slist => {"$(sys.policy_hub)"};
classes:
### Enable special features policies. Set to "any" to enable.
# Auto-load files in "services/autorun" and run bundles tagged "autorun".
# Disabled by default!
"services_autorun" expression => "!any";
# Internal CFEngine log files rotation
"cfengine_internal_rotate_logs" expression => "any";
# Disable agent email output
"cfengine_internal_agent_email" expression => "any";
# Transfer policies and binaries with encryption
# you can also request it from the command line with
# -Dcfengine_internal_encrypt_transfers
# NOTE THAT THIS CLASS ALSO NEEDS TO BE SET IN update.cf
"cfengine_internal_encrypt_transfers" expression => "!any";
# Purge policies that don't exist on the server side.
# you can also request it from the command line with
# -Dcfengine_internal_purge_policies
# NOTE THAT THIS CLASS ALSO NEEDS TO BE SET IN update.cf
"cfengine_internal_purge_policies" expression => "!any";
# Class defining which versions of cfengine are (not) supported
# by this policy version.
# Also note that this policy will only be run on enterprise policy_server
"postgresql_maintenance_supported" expression => "policy_server.enterprise.!(cfengine_3_4|cfengine_3_5)";
# Allow the hub to edit sudoers in order for the Apache user to
# run passwordless sudo cf-runagent. Enable this if you want the
# Mission Portal to be able to troubleshoot failed Design Center
# sketch activations on a host.
"cfengine_internal_sudoers_editing_enable" expression => "!any";
# Class defining which versions of cfengine are (not) supported
# by this policy version.
# Also note that this policy will only be run on enterprise policy_server
"postgresql_maintenance_supported"
expression => "(policy_server.enterprise.!cfengine_3_5.!enable_cfengine_enterprise_hub_ha)|(policy_server.enterprise.enable_cfengine_enterprise_hub_ha.hub_active)";
# This class is for PosgreSQL maintenance
# pre-defined to every Sunday at 2 a.m.
# This can be changed later on.
"postgresql_full_maintenance" expression => "postgresql_maintenance_supported.Sunday.Hr02.Min00_05";
# Vacuum monitoring_mg table every day (except Sunday, when vacuum is run on entire db)
# Required only by CFEngine Enterprise Hub 3.6.0 and 3.6.1
"postgresql_monitoring_maintenance" expression => "postgresql_maintenance_supported.!Sunday.Hr02.Min00_05.(cfengine_3_6_0|cfengine_3_6_1)";
# Run vacuum job on database
# pre-defined to every night except Sunday when full cleanup is executed.
"postgresql_vacuum" expression => "postgresql_maintenance_supported.!Sunday.Hr02.Min00_05";
# Enable CFEngine Enterprise HA Policy
"enable_cfengine_enterprise_hub_ha" expression => "!any";
#"enable_cfengine_enterprise_hub_ha" expression => "enterprise_edition";
# Enable cleanup of agent report diffs when they exceed
# `def.max_client_history_size`
"enable_cfe_internal_cleanup_agent_reports" -> { "cf-hub", "CFEngine Enterprise" }
expression => "enterprise_edition",
comment => "If reports are not collected for an extended period of time
the disk may fill up or cause additional collection
issues.";
}
bundle common inventory_control
{
vars:
"dmidecoder" string => "/usr/sbin/dmidecode";
"lldpctl_exec" string => "/usr/bin/lldpctl";
"lsb_exec" string => "/usr/bin/lsb_release";
"mtab" string => "/etc/mtab";
"proc" string => "/proc";
classes:
# setting this disables all the inventory modules except package_refresh
"disable_inventory" expression => "!any";
# disable specific inventory modules below
# by default disable the LSB inventory if the general inventory
# is disabled or the binary is missing. Note that the LSB
# binary is typically not very fast.
"disable_inventory_lsb" expression => "disable_inventory";
"disable_inventory_lsb" not => fileexists($(lsb_exec));
# by default disable the dmidecode inventory if the general
# inventory is disabled or the binary does not exist. Note that
# typically this is a very fast binary.
"disable_inventory_dmidecode" expression => "disable_inventory";
"disable_inventory_dmidecode" not => fileexists($(dmidecoder));
# by default disable the LLDP inventory if the general inventory
# is disabled or the binary does not exist. Note that typically
# this is a reasonably fast binary but still may require network
# I/O.
"disable_inventory_LLDP" expression => "disable_inventory";
"disable_inventory_LLDP" not => fileexists($(lldpctl_exec));
# by default run the package inventory refresh every time, even
# if disable_inventory is set
"disable_inventory_package_refresh" expression => "!any";
# by default disable the mtab inventory if the general inventory
# is disabled or $(mtab) is missing. Note that this is very
# fast.
"disable_inventory_mtab" expression => "disable_inventory";
"disable_inventory_mtab" not => fileexists($(mtab));
# by default disable the fstab inventory if the general
# inventory is disabled or $(sys.fstab) is missing. Note that
# this is very fast.
"disable_inventory_fstab" expression => "disable_inventory";
"disable_inventory_fstab" not => fileexists($(sys.fstab));
# by default disable the proc inventory if the general
# inventory is disabled or /proc is missing. Note that
# this is typically fast.
"disable_inventory_proc" expression => "disable_inventory";
"disable_inventory_proc" not => isdir($(proc));
# by default don't run the CMDB integration every time, even if
# disable_inventory is not set
"disable_inventory_cmdb" expression => "any";
reports:
verbose_mode.disable_inventory::
"$(this.bundle): All inventory modules disabled";
verbose_mode.!disable_inventory_lsb::
"$(this.bundle): LSB module enabled";
verbose_mode.!disable_inventory_dmidecode::
"$(this.bundle): dmidecode module enabled";
verbose_mode.!disable_inventory_LLDP::
"$(this.bundle): LLDP module enabled";
verbose_mode.!disable_inventory_mtab::
"$(this.bundle): mtab module enabled";
verbose_mode.!disable_inventory_fstab::
"$(this.bundle): fstab module enabled";
verbose_mode.!disable_inventory_proc::
"$(this.bundle): proc module enabled";
verbose_mode.!disable_inventory_package_refresh::
"$(this.bundle): package_refresh module enabled";
verbose_mode.!disable_inventory_cmdb::
"$(this.bundle): CMDB module enabled";
}
How it works
def.cf
has some crucial settings used by the rest of CFEngine. It's
expected that users will edit it but won't normally change the rest of
the masterfiles except in services
or if they know it's necessary.
This is a simple CFEngine policy, so read on for configuring it.
def.cf configuration
domain
Set your domain
to the right value. By default it's used for mail
and to deduce your file access ACLs.
acl
The acl
is crucial. This is used by every host, not just the
policy server. Make sure you only allow hosts you want to allow.
trustkeysfrom
trustkeysfrom
tells the policy server from which IPs it should accept
connections even if the host's key is unknown, trusting it at connect
time. This is only useful to be open during for bootstrapping these
hosts. As the comments say, empty it after your hosts have been
bootstrapped to avoid unpleasant surprises.
services_autorun
Off by default.
Turn this on (set to any
) to auto-load files in services/autorun
and run bundles found that are tagged autorun
. Here's a simple
example of such a bundle in services/autorun/hello.cf
:
bundle agent hello_world_autorun
{
meta:
"tags" slist => { "autorun" };
reports:
verbose_mode::
"$(this.bundle): Hello, this is an automatically loaded bundle";
}
cfengine_internal_rotate_logs
On by default. Rotates CFEngine's own logs. Here is the
cfe_internal_log_rotation
bundle implementation:
bundle agent cfe_internal_log_rotation
{
methods:
cfengine_internal_rotate_logs::
# CFEngine generates internal log files that need to be rotated.
# Have a look at def.cf to enable rotation of these files
"rotate_promise_summary" usebundle => logrotate("@(def.cfe_log_files)", "1m", "10"),
comment => "Rotate log files once their size reach 1MB, keep 10 versions";
"rotate_outputs" usebundle => prunedir("@(def.cfe_log_dirs)", "30"),
comment => "Delete outputs/* files older than 30 days";
}
cfengine_internal_agent_email
On by default. Enables agent email output from cf-execd
.
cfengine_internal_encrypt_transfers
Duplicate of the one in update.cf
. They should be set in unison or
you will get unexpected behavior.
cfengine_internal_purge_policies
Duplicate of the one in update.cf
. They should be set in unison or
you will get unexpected behavior.
cfengine_internal_preserve_permissions
Duplicate of the one in update.cf
. They should be set in unison or
you will get unexpected behavior.
cfengine_internal_sudoers_editing_enable
Off by default. Only used on the CFEngine Enterprise hub.
Turn this on (set to any
) to allow the hub to edit sudoers in order
for the Apache user to run passwordless sudo cf-runagent (part of
Mission Portal troubleshooting).
def.cf inventory control
The inventory is a cool new feature in 3.6. You can disable pieces of it (inventory modules) or the whole thing if you wish.
disable_inventory
This class is off by default (meaning the inventory is on by default). Here's the master switch to disable all inventory modules.
disable_inventory_lsb
LSB is the Linux Standard Base, see https://wiki.linuxfoundation.org/en/LSB
By default, this class is turned off (and the module is on) if the LSB
executable /usr/bin/lsb_release
can be found. This inventory module
will populate inventory reports and variables for you with LSB
details. For details, see LSB
disable_inventory_dmidecode
By default, this class is turned off (and the module is on) if the
executable /usr/sbin/dmidecode
can be found. This inventory module
will populate inventory reports and variables for you. For details,
see DMI decoding
disable_inventory_LLDP
LLDP is a protocol for Link Layer Discovery. See http://en.wikipedia.org/wiki/Link_Layer_Discovery_Protocol
By default, this class is turned off (and the module is on) if the
executable /usr/bin/lldpctl
can be found. This inventory module will
populate variables for you. For details, see LLDP
disable_inventory_package_refresh
By default, this class is turned off (and the module is on). This inventory module will populate the installed packages for you. On CFEngine Enterprise, the available packages will also be populated.
disable_inventory_mtab
By default, this class is turned off (and the module is on) if
/etc/mtab
exists. This inventory module will populate variables for
you based on the mounted filesystems. For details, see mtab
disable_inventory_fstab
By default, this class is turned off (and the module is on) if
$(sys.fstab)
(usually /etc/fstab
or /etc/vfstab
) exists. This
inventory module will populate variables for you based on the defined
filesystems. For details, see fstab
disable_inventory_proc
By default, this class is turned off (and the module is on) if /proc
is a directory. This inventory module will populate variables for you
from some of the contents of /proc
. For details, see procfs
disable_inventory_cmdb
By default, this class is turned on (and the module is off).
Turn this on (set to any
) to allow each client to load a me.json
file from the server and load its contents. For details, see CMDB
promises.cf
How it works
promises.cf
is your main run file. Keep referring to your
installation's promises.cf
as you read this.
promises.cf
is the first file that cf-agent
with no arguments will
try to look for. So whenever you see cf-agent
with no flile
parameter, read it as "run my promises.cf
".
It should contain all of the basic configuration
settings, including a list of other files to include. In normal
operation, it must also have a bundlesequence
.
promises.cf configuration
bundlesequence
The bundlesequence
acts like the 'genetic makeup' of the
configuration. Edit the bundlesequence
to add any bundles you have
defined, or are pre-defined. Consider using the services_autorun
facility so you don't have to edit this setting at all.
BEWARE THAT ONLY VALID (KNOWN) BUNDLES CAN BE ADDED.
By default, the inventory modules, then internal hub modules, then
Design Center sketches, then the autorun services, and finally
internal management bundles are in the bundlesequence
.
In a large configuration, you might want to have a different
bundlesequence
for different classes of host, so that you can build
a complete system like a check-list from different combinations of
building blocks. You can construct different lists by composing them
from other lists, or you can use methods promises as an alternative
for composing bundles for different classes. This is an advanced topic
and a risky area (if you get it wrong, your policies will not
validate) so make sure you test your changes carefully!
inputs
In order to find bundles, CFEngine needs to know where to look. This
list defines what files are needed. Note there are several dynamic
entries here, coming from other bundles. CFEngine will keep evaluating
the inputs
and bundlesequence
until all the bundles are found and
resolved.
Make sure to add any of your own services
files here if you don't
use the services_autorun
facility, to ensure the bundles in them are
found.
failsafe.cf
The failsafe.cf
file ensures that your system can survive errors and
can upgrade gracefully to new versions even when mistakes are made.
It's literally a failsafe if promises.cf
and update.cf
should
fail.
This file is generated during the bootstrapping process, and should
normally never be changed. The only job of failsafe.cf
is to execute
the update bundle in a “standalone” context should there be a syntax
error somewhere in the main set of promises. In this way, if a client
machine's policies are ever corrupted after downloading erroneous
policy from a server, that client will have a failsafe method for
downloading a corrected policy once it becomes available on the
server. Note that by “corrupted” and “erroneous” we typically mean
“broken via administrator error” - mistakes happen, and the
failsafe.cf
file is CFEngine's way of being prepared for that
eventuality.
If you ever change failsafe.cf
(or when you upgrade CFEngine), make
sure the old and the new CFEngine can successfully parse and execute
this file. If not, you risk losing control of your system (that is, if
CFEngine cannot successfully execute this policy file, it has no
failsafe/fallback mechanism for distributing new policy files).
Some general rules (but again, note you can completely break your
CFEngine installation by editing failsafe.cf
):
- Upgrade the software first, then add new features to the configuration.
- Never use advanced features in the failsafe or update file.
- Avoid using library code (including any bodies from
stdlib.cf
or the files it includes). Copy/paste any bodies you need using a unique name that does not collide with a name in library (we recommend simply adding the prefix “u_”). This may mean that you create duplicate functionality, but that is okay in this case to ensure a 100% functioning standalone update process). The promises which manage the update process should not have any dependencies on any other files.
CFEngine will fail-over to the failsafe.cf
configuration if it is
unable to read or parse the contents successfully. That means that any
syntax errors you introduce (or any new features you utilize in a
configuration) will cause a fail-over, because the parser will not be
able to interpret the policy. If the failover is due to the use of new
features, they will not parse until the software itself has been
updated (so we recommend that you always update CFEngine before
updating policy to use new features). If you accidentally cause a bad
(i.e., unparseable) policy to be distributed to client machines, the
failsafe.cf
policy on those machines will run (and will eventually
download a working policy, once you fix it on the policy host).
Further structure
cfe_internal
: internal CFEngine policies you shouldn't modify or you will get unexpected behaviorcontrols
: configuration of components, e.g. thecf-agent
orcf-serverd
, beyond whatdef.cf
can offerdef.cf
: defaults you can and should configure, see aboveinventory
: inventory modules (loaded before anything else to discover facts about the system) live here; see abovelib
: main library directory. You'll see3.5
and3.6
and3.7
under it. These are the supported versions for masterfiles backwards compatibility.promises.cf
: main policy, you will need to configure this, see aboveservices
: your site's policies go hereservices_autorun
: see abovesketches
: Design Center installations use this; do not touch or you will get unexpected behaviorupdate
andupdate.cf
: functionality for updating inputs and CFEngine itself, see above. You shouldn't modify files underupdate
or you will get unexpected behavior.
cf_promises_validated
Several CFEngine components that read policy (e.g. cf-agent
, cf-execd
, cf-serverd
) run cf-promises
to validate the syntax of their input files before actually running the policy. To illustrate this, if cf-promises
runs every 5 minutes then there will be 12 checks occurring every hour, 24 hours a day, 7 days a week -- a total of 2016 possible validation checks. Each of those individual validation sessions can take some number of seconds to perform depending on the system, scale, circumstances and configuration.
Starting with CFEngine 3.1.2, the outcome of every run of cf-promises
was cached, which lets agents skip the validation of input files that have not changed since the previous run.
Starting with CFEngine 3.6, outcome on both hosts and hubs is stored in the file $(sys.workdir)/masterfiles/cf_promises_validated
(usually sys.workdir
is /var/cfengine
). The file can be created by cf-agent
after it has successfully verified the policy with cf-promises
. The file can also be created by a user with cf-promises -T DIRECTORY
which is useful for validating an entire directory.
When the hash content of any file under WORKDIR/inputs
changes, and validates to be syntactically correct, then a timestamp in cf_promises_validated
is updated. If not, the run of cf-promises
is skipped and, at the same time, the cf-execd, cf-serverd and cf-monitord daemons will not reload the policy unless cf_promises_validated
has an updated timestamp, which cf-agent
will normally take care of.
In the default installation, the masterfiles are populated automatically on the policy server and you can even auto-deploy them from a version control system.
You should configure the masterfiles as described above. Leaving them at their default settings may expose your masterfiles or worse, especially the cf-serverd ACL settings. If you are not sure of the terms used below or what it all means, come back to this page after you've learned about writing policy and the CFEngine syntax.
CFEngine 3 Inventory Modules
The CFEngine 3 inventory modules are pieces of CFEngine policy that
are loaded and used by the promises.cf
mechanism in order to
inventory the system.
CFEngine Enterprise has specific functionality to show and use inventory data, but users of the Community Version can use them as well locally on each host.
How It Works
The inventory modules are called in promises.cf
:
body common control
{
bundlesequence => {
# Common bundle first (Best Practice)
inventory_control,
@(inventory.bundles),
...
As you see, this calls the inventory_control
bundle, and then each
bundle in the list inventory.bundles
. That list is built in the
top-level common inventory
bundle, which will load the right things
for some common cases. The any.cf
inventory module is always loaded;
the rest are loaded if they are appropriate for the platform. For
instance, Debian systems will load debian.cf
and linux.cf
and
lsb.cf
but may load others as needed.
The effect for users is that the right inventory modules will be loaded and evaluated.
The inventory_control
bundle lives in def.cf
and defines what
inventory modules should be disabled. You can simply set
disable_inventory
to avoid the whole system, or you can look for the
disable_inventory_xyz
class to disable module xyz
.
Any inventory module works the same way, by doing some discovery work
and then tagging its classes and variables with the report
or
inventory
tags. For example:
vars:
"ports" slist => { @(mon.listening_ports) },
meta => { "inventory", "attribute_name=Ports listening" };
This defines a reported attribute "Ports listening" which contains a list of strings representing the listening ports. More on this in a second.
Your Very Own Inventory Module
The good news is, writing an inventory module is incredibly easy.
They are just CFEngine bundles. You can see a simple one that collects
the listening ports in any.cf
:
bundle agent cfe_autorun_inventory_listening_ports
# @brief Inventory the listening ports
#
# This bundle uses `mon.listening_ports` and is always enabled by
# default, as it runs instantly and has no side effects.
{
vars:
"ports" slist => { @(mon.listening_ports) },
meta => { "inventory", "attribute_name=Ports listening" };
}
Well, the slist copy is a CFEngine detail (we get the listening ports
from the monitoring daemon), so just assume that the data is correct.
What's important is the second line that starts with
meta
. That
defines metadata for the promise that CFEngine will use to determine
that this data is indeed inventory data and should be reported to the
CFEngine Enterprise Hub.
That's it. Really. The comments are optional but nice to have. You
don't have to put your new bundle in a file under the inventory
directory, either. The variables and classes can be declared anywhere
as long as they have the right tags. So you can use the services
directory or whatever else makes sense to you.
CFEngine Enterprise vs. Community
In CFEngine Enterprise, the reported data is aggregated in the hub and reported across the whole host population.
In CFEngine Community, users can use the classesmatching()
and
variablesmatching()
functions to collect all the inventory variables
and classes and report them in other ways.
Implementation Best Practice for CFEngine Enterprise
It is important that inventory variables and classes are continually defined. Only inventory variables and classes defined during the last reported run are available for use by the inventory reporting interface.
Inventory items that change frequently can create a burden on the Enterprise reporting infrastructure. Generally, inventory attributes should change infrequently.
If you wish to inventory attributes that frequently change or are expensive to discover consider implementing a sample interval and caching mechanism.
What Modules Are Available?
As soon as you use the promises.cf
provided in the parent directory,
quite a few inventory modules will be enabled (if appropriate for your
system). Here's the list of modules and what they provide. Note they
are all enabled by code in def.cf
as explained above.
LSB
- lives in:
lsb.cf
- applies to: LSB systems (most Linux distributions, basically)
- runs:
lsb_release -a
- sample data:
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty
- provides:
- classes
lsb_$(os)
,lsb_$(os)_$(release)
,lsb_$(os)_$(codename)
- variables:
inventory_lsb.os
(Distributor ID),inventory_lsb.codename
,inventory_lsb.release
,inventory_lsb.flavor
,inventory_lsb.description
- classes
- implementation:
bundle agent inventory_lsb
{
classes:
"have_lsb" expression => fileexists($(lsb_exec));
"_inventory_lsb_found" expression => regcmp("^[1-9][0-9]*$", $(dim)),
scope => "namespace";
_inventory_lsb_found::
"lsb_$(os)" expression => "any",
comment => "LSB Distributor ID",
depends_on => { "inventory_lsb_os" },
scope => "namespace",
meta => { "inventory", "attribute_name=none" };
"lsb_$(os)_$(release)" expression => "any",
comment => "LSB Distributor ID and Release",
depends_on => { "inventory_lsb_os", "inventory_lsb_release" },
scope => "namespace",
meta => { "inventory", "attribute_name=none" };
"lsb_$(os)_$(codename)" expression => "any",
comment => "LSB Distributor ID and Codename",
depends_on => { "inventory_lsb_os", "inventory_lsb_codename" },
scope => "namespace",
meta => { "inventory", "attribute_name=none" };
vars:
"lsb_exec" string => "$(inventory_control.lsb_exec)";
have_lsb::
"data" string => execresult("$(lsb_exec) -a", "noshell");
"dim" int => parsestringarray(
"lsb",
$(data),
"\s*#[^\n]*",
"\s*:\s+",
"15",
"4095"
);
_inventory_lsb_found::
"lsb_keys" slist => getindices("lsb");
"os" string => canonify("$(lsb[Distributor ID][1])"),
handle => "inventory_lsb_os",
comment => "LSB-provided OS name",
meta => { "inventory", "attribute_name=none" };
"codename" string => canonify("$(lsb[Codename][1])"),
handle => "inventory_lsb_codename",
comment => "LSB-provided OS code name",
meta => { "inventory", "attribute_name=none" };
"release" string => "$(lsb[Release][1])",
handle => "inventory_lsb_release",
comment => "LSB-provided OS release",
meta => { "inventory", "attribute_name=none" };
"flavor" string => canonify("$(lsb[Distributor ID][1])_$(lsb[Release][1])"),
handle => "inventory_lsb_flavor",
comment => "LSB-provided OS flavor",
meta => { "inventory", "attribute_name=none" };
"description" string => "$(lsb[Description][1])",
handle => "inventory_lsb_description",
comment => "LSB-provided OS description",
meta => { "inventory", "attribute_name=none" };
reports:
inform_mode._inventory_lsb_found::
"$(this.bundle): OS = $(os), codename = $(codename), release = $(release), flavor = $(flavor), description = $(description)";
verbose_mode._inventory_lsb_found::
"$(this.bundle): got $(dim) LSB keys";
"$(this.bundle): prepared LSB key $(lsb_keys) = '$(lsb[$(lsb_keys)][1])'";
}
- sample output:
% cf-agent -KI -binventory_control,inventory_lsb
R: inventory_lsb: OS = Ubuntu, codename = trusty, release = 14.04, flavor = Ubuntu_14_04, description = Ubuntu 14.04 LTS
SUSE
- lives in:
suse.cf
- applies to: SUSE Linux
- provides classes:
suse_pure
andsuse_derived
- implementation:
bundle common inventory_suse
{
classes:
"suse_pure" expression => "(sles|sled).!opensuse",
comment => "pure SUSE",
meta => { "inventory", "attribute_name=none" };
"suse_derived" expression => "opensuse.!suse_pure",
comment => "derived from SUSE",
meta => { "inventory", "attribute_name=none" };
}
Debian
- lives in:
debian.cf
- applies to: Debian and its derivatives
- provides:
- variables:
inventory_debian.mint_release
andinventory_debian.mint_codename
- classes:
debian_pure
,debian_derived
,linuxmint
,lmde
,linuxmint_$(mint_release)
,linuxmint_$(mint_codename)
,$(mint_codename)
- variables:
- implementation:
bundle common inventory_debian
{
vars:
has_lsb_release::
"lsb_release_info" string => readfile("/etc/lsb-release","256"),
comment => "Read more OS info" ;
has_etc_linuxmint_info::
"linuxmint_info" string => readfile("/etc/linuxmint/info","1024"),
comment => "Read Linux Mint specific info" ;
"lm_info_count"
int => parsestringarray("mint_info", # array to populate
"$(linuxmint_info)", # data to parse
"\s*#[^\n]*", # comments
"=", # split
100, # maxentries
2048) ; # maxbytes
"mint_release" string => "$(mint_info[RELEASE][1])" ;
"mint_codename" string => "$(mint_info[CODENAME][1])" ;
classes:
any::
"debian_derived_evaluated"
scope => "bundle",
or => { "has_os_release", "has_lsb_release", "has_etc_linuxmint_info" } ;
"linuxmint"
expression => "has_etc_linuxmint_info",
comment => "this is a Linux Mint system, of some sort",
meta => { "inventory", "attribute_name=none" } ;
has_lsb_release::
"linuxmint"
expression => regcmp("(?ms).*^DISTRIB_ID=LinuxMint$.*", "$(lsb_release_info)"),
comment => "this is a Linux Mint system, of some sort",
meta => { "inventory", "attribute_name=none" } ;
linuxmint.has_os_release::
"lmde"
expression => regcmp('(?ms).*^NAME="Linux Mint LMDE"$.*', "$(inventory_linux.os_release_info)"),
comment => "this is a Linux Mint Debian Edition",
meta => { "inventory", "attribute_name=none", "derived-from=inventory_linux.os_release_info" } ;
linuxmint.has_lsb_release::
"lmde"
expression => regcmp('(?ms).*^DISTRIB_DESCRIPTION="LMDE.*', "$(lsb_release_info)"),
comment => "this is a Linux Mint Debian Edition",
meta => { "inventory", "attribute_name=none", "derived-from=inventory_debian.lsb_release_info" } ;
has_etc_linuxmint_info::
"lmde"
expression => regcmp('(?ms).*^DESCRIPTION="LMDE.*',"$(linuxmint_info)"),
comment => "this is a Linux Mint Debian Edition",
meta => { "inventory", "attribute_name=none", "derived-from=inventory_debian.linuxmint_info" } ;
debian_derived_evaluated.has_etc_linuxmint_info.!lmde::
# These need to be evaluated only after debian_derived_evaluated is defined
# to ensure that the mint_info array has been evaluated as well.
# Failing to do that will create meaningless classes
# On non-LMDE Mint systems, this will create classes like, e.g.:
# linuxmint_14, nadia, linuxmint_nadia
"linuxmint_$(mint_release)" expression => "any",
meta => { "inventory", "attribute_name=none" } ;
"$(mint_codename)" expression => "any",
meta => { "inventory", "attribute_name=none" } ;
"linuxmint_$(mint_codename)" expression => "any",
meta => { "inventory", "attribute_name=none" } ;
debian_derived_evaluated::
"debian_pure" expression => "debian.!(ubuntu|linuxmint)",
comment => "pure Debian",
meta => { "inventory", "attribute_name=none" };
"debian_derived" expression => "debian.!debian_pure",
comment => "derived from Debian",
meta => { "inventory", "attribute_name=none" };
any::
"has_lsb_release" expression => fileexists("/etc/lsb-release"),
comment => "Check if we can get more info from /etc/lsb-release";
"has_etc_linuxmint_info" expression => fileexists("/etc/linuxmint/info"),
comment => "If this is a Linux Mint system, this *could* be available";
}
Red Hat
- lives in:
redhat.cf
- applies to: Red Hat and its derivatives
- provides classes:
redhat_pure
,redhat_derived
- implementation:
bundle common inventory_redhat
{
classes:
"redhat_pure" expression => "redhat.!centos.!oracle",
comment => "pure Red Hat",
meta => { "inventory", "attribute_name=none" };
"redhat_derived" expression => "redhat.!redhat_pure",
comment => "derived from Red Hat",
meta => { "inventory", "attribute_name=none" };
}
Windows
- lives in:
windows.cf
Mac OS X
- lives in:
macos.cf
Generic (unknown OS)
- lives in:
generic.cf
(seeany.cf
for generally applicable inventory modules)
LLDP
- lives in:
any.cf
- runs
inventory_control.lldpctl_exec
through a Perl filter - provides variables:
cfe_autorun_inventory_LLDP.K
for eachK
returned by the LLDB executable - implementation:
bundle agent cfe_autorun_inventory_LLDP
{
classes:
"disable_inventory_LLDP" not => fileexists($(inventory_control.lldpctl_exec));
commands:
!disable_inventory_LLDP::
"$(inventory_control.lldpctl_exec) | perl -n -e 'my ($k, $v) = m/([^=]+)=(.*)/; $k =~ s/\W/_/g; print \"=$k=$v\";"
classes => kept_successful_command,
module => "true";
}
mtab
- lives in:
any.cf
- parses:
/etc/mtab
- provides classes:
have_mount_FSTYPE
andhave_mount_FSTYPE_MOUNTPOINT
- implementation:
bundle agent cfe_autorun_inventory_mtab
{
vars:
have_mtab::
"mount_count" int => readstringarrayidx("mounts",
$(inventory_control.mtab),
"\s*#[^\n]*",
"\s+",
500,
50000);
"idx" slist => getindices("mounts");
classes:
"have_mtab" expression => fileexists($(inventory_control.mtab));
# define classes like have_mount_ext4__var for a ext4 /var mount
"have_mount_$(mounts[$(idx)][2])_$(mounts[$(idx)][1])"
expression => "any",
scope => "namespace";
# define classes like have_mount_ext4 if there is a ext4 mount
"have_mount_$(mounts[$(idx)][2])"
expression => "any",
scope => "namespace";
reports:
verbose_mode::
"$(this.bundle): we have a $(mounts[$(idx)][2]) mount under $(mounts[$(idx)][1])";
}
- sample output (note this is verbose mode with
-v
because there's a lot of output):
% cf-agent -Kv -binventory_control,cfe_autorun_inventory_mtab|grep 'cfe_autorun_inventory_mtab: we have'
R: cfe_autorun_inventory_mtab: we have a ext4 mount under /
...
R: cfe_autorun_inventory_mtab: we have a cgroup mount under /sys/fs/cgroup/systemd
R: cfe_autorun_inventory_mtab: we have a tmpfs mount under /run/shm
fstab
- lives in:
any.cf
- parses:
sys.fstab
- provides classes:
have_fs_FSTYPE
have_fs_MOUNTPOINT
andhave_fs_FSTYPE_MOUNTPOINT
- implementation:
bundle agent cfe_autorun_inventory_fstab
{
vars:
have_fstab::
"mount_count" int => readstringarrayidx("mounts",
$(sys.fstab),
"\s*#[^\n]*",
"\s+",
500,
50000);
"idx" slist => getindices("mounts");
classes:
"have_fstab" expression => fileexists($(sys.fstab));
# define classes like have_fs_ext4__var for a ext4 /var entry
"have_fs_$(mounts[$(idx)][2])_$(mounts[$(idx)][1])"
expression => "any",
scope => "namespace";
# define classes like have__var for a /var entry
"have_fs_$(mounts[$(idx)][1])"
expression => "any",
scope => "namespace";
# define classes like have_fs_ext4 if there is a ext4 entry
"have_fs_$(mounts[$(idx)][2])"
expression => "any",
scope => "namespace";
reports:
verbose_mode::
"$(this.bundle): we have a $(mounts[$(idx)][2]) fstab entry under $(mounts[$(idx)][1])";
}
- sample output (note this is verbose mode with
-v
because there's a LOT of output):
% cf-agent -Kv -binventory_control,cfe_autorun_inventory_fstab|grep 'cfe_autorun_inventory_fstab: we have'
R: cfe_autorun_inventory_fstab: we have a ext4 fstab entry under /
R: cfe_autorun_inventory_fstab: we have a cifs fstab entry under /backups/load
R: cfe_autorun_inventory_fstab: we have a auto fstab entry under /mnt/cdrom
CMDB
- lives in:
any.cf
- parses:
me.json
(which is copied from the policy server; see implementation) - provides classes:
CLASS
for each CLASS found under theclasses
key in the JSON data - provides variables:
inventory_cmdb_load.VARNAME
for each VARNAME found under thevars
key in the JSON data - implementation:
bundle agent inventory_cmdb_load(file)
{
classes:
"have_cmdb_data" expression => isvariable("cmdb");
"$(ckeys)" expression => "any", scope => "namespace";
vars:
"cmdb" data => readjson($(file), "999999");
"cmdb_string" string => format("%S", cmdb);
"bkeys" slist => getindices("cmdb[vars]");
"vkeys_$(bkeys)" slist => getindices("cmdb[vars][$(bkeys)]");
"$(vkeys_$(bkeys))" string => nth("cmdb[vars][$(bkeys)]", $(vkeys));
"ckeys" slist => getindices("cmdb[classes]");
reports:
inform_mode.have_cmdb_data::
"$(this.bundle): Got CMDB data from $(file): $(cmdb_string)";
verbose_mode.have_cmdb_data::
"$(this.bundle): Got CMDB key = $(vkeys_$(bkeys)), CMDB value = $((vkeys_$(bkeys)))";
"$(this.bundle): Got CMDB class = $(ckeys)";
inform_mode.!have_cmdb_data::
"$(this.bundle): Could not read the CMDB data from $(file)";
}
DMI decoding
- lives in:
any.cf
- runs:
dmidecode
- provides variables:
cfe_autorun_inventory_dmidecode.dmi[K]
for each key K in thedmidecode
output - implementation:
bundle agent cfe_autorun_inventory_dmidecode
{
vars:
"dmidefs" data => parsejson('
{
"bios-vendor": "BIOS vendor",
"bios-version": "BIOS version",
"system-serial-number": "System serial number",
"system-manufacturer": "System manufacturer",
"system-version": "System version",
"processor-version": "CPU model"
}');
- sample output (sudo is needed to access the DMI):
% sudo /var/cfengine/bin/cf-agent -KI -binventory_control,cfe_autorun_inventory_dmidecode
R: cfe_autorun_inventory_dmidecode: Obtained BIOS vendor = 'Intel Corp.'
R: cfe_autorun_inventory_dmidecode: Obtained BIOS version = 'BLH6710H.86A.0146.2013.1555.1888'
R: cfe_autorun_inventory_dmidecode: Obtained System serial number = ''
R: cfe_autorun_inventory_dmidecode: Obtained System manufacturer = ''
R: cfe_autorun_inventory_dmidecode: Obtained System version = ''
R: cfe_autorun_inventory_dmidecode: Obtained CPU model = 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz'
Listening ports
- lives in:
any.cf
- provides variables:
cfe_autorun_inventory_listening_ports.ports
as a copy ofmon.listening_ports
- implementation:
bundle agent cfe_autorun_inventory_listening_ports
{
vars:
"ports" slist => sort( "mon.listening_ports", "int"),
meta => { "inventory", "attribute_name=Ports listening" };
}
Disk space
- lives in:
any.cf
- provides variables:
cfe_autorun_inventory_disk.free
as a copy ofmon.value_diskfree
- implementation:
bundle agent cfe_autorun_inventory_disk
{
vars:
enterprise::
"free" string => "$(mon.value_diskfree)",
meta => { "inventory", "attribute_name=Disk free (%)" };
}
Available memory
- lives in:
any.cf
- provides variables:
cfe_autorun_inventory_memory.free
as a copy ofmon.value_mem_free
andcfe_autorun_inventory_memory.total
as a copy ofmon.value_mem_total
- implementation:
bundle agent cfe_autorun_inventory_memory
{
vars:
enterprise::
# due to a Windows issue this is set to 0 there for now
"total" string => ifelse("windows", "0",
$(mon.value_mem_total)),
meta => { "inventory", "attribute_name=Memory size (MB)" };
"free" string => ifelse("windows", "0",
$(mon.value_mem_free)),
meta => { "report" };
}
Load average
- lives in:
any.cf
- provides variables:
cfe_autorun_inventory_loadaverage.value
as a copy ofmon.value_loadavg
- implementation:
bundle agent cfe_autorun_inventory_loadaverage
{
vars:
enterprise::
"value" string => "$(mon.value_loadavg)",
meta => { "report" };
}
procfs
- lives in:
any.cf
- parses:
consoles
,cpuinfo
,modules
,partitions
,version
- provides variables:
cfe_autorun_inventory_proc.console_count
,cfe_autorun_inventory_proc.cpuinfo[K]
for each CPU info key,cfe_autorun_inventory_proc.paritions[K]
for each partition key - provides classes:
_have_console_CONSOLENAME
,have_module_MODULENAME
- implementation:
bundle agent cfe_autorun_inventory_proc
{
vars:
"basefiles" slist => { "consoles", "cpuinfo", "modules", "partitions", "version" };
"files[$(basefiles)]" string => "$(inventory_control.proc)/$(basefiles)";
_have_proc_consoles::
"console_count" int => readstringarrayidx("consoles",
"$(files[consoles])",
"\s*#[^\n]*",
"\s+",
500,
50000);
"console_idx" slist => getindices("consoles");
_have_proc_modules::
"module_count" int => readstringarrayidx("modules",
"$(files[modules])",
"\s*#[^\n]*",
"\s+",
2500,
250000);
"module_idx" slist => getindices("modules");
_have_proc_cpuinfo::
# this will extract all the keys in one bunch, so you won't get
# detailed info for processor 0 for example
"cpuinfo_count" int => readstringarrayidx("cpuinfo_array",
"$(files[cpuinfo])",
"\s*#[^\n]*",
"\s*:\s*",
500,
50000);
"cpuinfo_idx" slist => getindices("cpuinfo_array");
"cpuinfo[$(cpuinfo_array[$(cpuinfo_idx)][0])]" string => "$(cpuinfo_array[$(cpuinfo_idx)][1])";
"cpuinfo_keys" slist => getindices("cpuinfo");
_have_proc_partitions::
"partitions_count" int => readstringarrayidx("partitions_array",
"$(files[partitions])",
"major[^\n]*",
"\s+",
500,
50000);
"partitions_idx" slist => getindices("partitions_array");
"partitions[$(partitions_array[$(partitions_idx)][4])]" string => "$(partitions_array[$(partitions_idx)][3])";
"partitions_keys" slist => getindices("partitions");
_have_proc_version::
"version" string => readfile("$(files[version])", 2048);
classes:
"have_proc" expression => isdir($(inventory_control.proc));
have_proc::
"_have_proc_$(basefiles)"
expression => fileexists("$(files[$(basefiles)])");
_have_proc_consoles::
"have_console_$(consoles[$(console_idx)][0])"
expression => "any",
scope => "namespace";
_have_proc_modules::
"have_module_$(modules[$(module_idx)][0])"
expression => "any",
scope => "namespace";
reports:
_have_proc_consoles.verbose_mode::
"$(this.bundle): we have console $(consoles[$(console_idx)][0])";
_have_proc_modules.verbose_mode::
"$(this.bundle): we have module $(modules[$(module_idx)][0])";
_have_proc_cpuinfo.verbose_mode::
"$(this.bundle): we have cpuinfo $(cpuinfo_keys) = $(cpuinfo[$(cpuinfo_keys)])";
_have_proc_partitions.verbose_mode::
"$(this.bundle): we have partitions $(partitions_keys) with $(partitions[$(partitions_keys)]) blocks";
_have_proc_version.verbose_mode::
"$(this.bundle): we have kernel version '$(version)'";
}
- sample output (note this is verbose mode with
-v
because there's a LOT of output):
% cf-agent -Kv -binventory_control,cfe_autorun_inventory_proc|grep 'cfe_autorun_inventory_proc: we have'
R: cfe_autorun_inventory_proc: we have console tty0
R: cfe_autorun_inventory_proc: we have module snd_seq_midi
...
R: cfe_autorun_inventory_proc: we have module ghash_clmulni_intel
R: cfe_autorun_inventory_proc: we have cpuinfo flags = fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
...
R: cfe_autorun_inventory_proc: we have cpuinfo model name = Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
R: cfe_autorun_inventory_proc: we have partitions sr0 with 1048575 blocks
...
R: cfe_autorun_inventory_proc: we have partitions sda with 468851544 blocks
R: cfe_autorun_inventory_proc: we have kernel version 'Linux version 3.11.0-15-generic (buildd@roseapple) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #25-Ubuntu SMP Thu Jan 30 17:22:01 UTC 2014'
Testing Policies
One of the practical advantages of CFEngine is that you can test it without the need for root or administrator privileges. This is useful if you are concerned about manipulating important system files, but naturally limits the possibilities for what CFEngine is able to do.
CFEngine operates with the notion of a work-directory. The default work
directory for the root
user is /var/cfengine
. For any other user, the work
directory lies in the user's home directory, named ~/.cfagent
.
CFEngine prefers you to keep certain files here. You should not resist this too strongly or you will make unnecessary trouble for yourself. The decision to have this 'known directory' was made to simplify a lot of configuration.
To test CFEngine as an ordinary user, do the following:
Copy the binaries into the work directory:
host$ mkdir -p ~/.cfagent/inputs
host$ mkdir -p ~/.cfagent/bin
host$ cp /var/cfengine/bin/cf-* ~/.cfagent/bin
host$ cp /var/cfengine/inputs/*.cf ~/.cfagent/inputs
You can test the software and play with configuration files by editing the
basic directly in the ~/.cfagent/inputs
directory. For example, try the
following:
host$ ~/.cfagent/bin/cf-promises
host$ ~/.cfagent/bin/cf-promises --verbose
This is always the way to start checking a configuration in CFEngine 3. If a
configuration does not pass this check/test, you will not be allowed to use
it, and cf-agent
will look for the file failsafe.cf
.
Language Concepts
There is only one grammatical form for statements in the language:
bundle bundle_type name
{
promise_type:
classes::
"promiser" -> { "promisee1", "promisee2", ... }
attribute_1 => value_1,
attribute_2 => value_2,
...
attribute_n => value_n;
}
In addition, CFEngine bodies can be defined and used as attribute values. Here's a real-life example of a body and its usage.
body edit_defaults no_backup
{
edit_backup => "false";
}
... and elsewhere, noting the attribute name matches the body type ...
files:
"myfile" edit_defaults => no_backup;
You can recognize everything in CFEngine from just those few concepts.
A declaration about the state we desire to maintain (e.g., the permissions or contents of a file, the availability or absence of a service, the (de)installation of a package).
A collection of promises.
A part of a promise which details and constrains its nature, possibly in separate and re-usable parts. Effectively a body is like a promise attribute that has several parameters.
CFEngine's boolean classifiers that describe context.
An association of the form "LVALUE represents RVALUE", where RVALUE may be a scalar value or a list of scalar values: a string, integer or real number.
This documentation about the language concepts introduces in addition
Syntax, identifiers and names
The CFEngine 3 language has a few simple rules:
- CFEngine built-in words, names of variables, bundles, body templates and classes may only contain the usual alphanumeric and underscore characters (
a-zA-Z0-9_
) - All other 'literal' data must be quoted.
Declarations of promise bundles in the form:
bundle agent-type identifier { ... }
where
agent-type
is the CFEngine component responsible for maintaining the promise.Declarations of promise body-parts in the form:
body constraint_type template_identifier { ... }
matching and expanding on a reference inside a promise of the form
constraint_type => template_identifier
attribute expressions in the body of a promise take the form
left-hand-side (CFEngine_word) => right-hand-side (user defined data).
This can take several forms:
cfengine_word => user_defined_template(parameters) user_defined_template builtin_function() "quoted literal scalar" { list }
In each of these cases, the right hand side is a user choice.
CFEngine uses many `constraint expressions' as part of the body of a promise. These take the form: left-hand-side (cfengine word) ‘=>’ right-hand-side (user defined data). This can take several forms:
cfengine_word => user_defined_template(parameters) user_defined_template builtin_function() "quoted literal scalar" { list }
In each of these cases, the right hand side is a user choice.
Filenames and Paths
Filenames in Unix-like operating systems use the forward slash '/' character for their directory separator. All references to file locations must be absolute pathnames in CFEngine, i.e. they must begin with a complete specification of which directory they are in or with a variable reference that resolves to that. For example:
/etc/passwd
/var/cfengine/masterfiles/distfile
$(sys.masterdir)/distfile # usually the same thing in 3.6
The only place where it makes sense to refer to a file without a complete directory specification is when searching through directories for different kinds of file, e.g. in pattern matching
leaf_name => { "tmp_.*", "output_file", "core" };
Here, one can write core without a path, because one is looking for any file of that name in a number of directories.
The Windows operating systems traditionally use a different filename convention. The following are all valid absolute file names under Windows:
c:\winnt
"c:\spaced name"
c:/winnt
/var/cfengine/inputs
//fileserver/share2/dir
The 'drive' name "C:" in Windows refers to a partition or device. Unlike Unix, Windows does not integrate these seamlessly into a single file-tree. This is not a valid absolute filename:
\var\cfengine\inputs
Paths beginning with a backslash are assumed to be win32 paths. They must begin with a drive letter or double-slash server name.
Note that in many cases, you have sys.inputdir
and other
Special Variables that work equally well on Windows and non-Windows
system.
Note in recent versions of Cygwin you can decide to use the
/cygdrive
to specify a path to windows file E.g
/cygdrive/c/myfile means c:\myfile or you can do it straight away
in CFEngine as c:\myfile
.
Promises
One concept in CFEngine should stand out from the rest as being the most important: promises. Everything else is just an abstraction that allows us to declare promises and model the various actors in the system.
Everything is a Promise
Everything in CFEngine 3 can be interpreted as a promise. Promises can be made about all kinds of different subjects, from file attributes, to the execution of commands, to access control decisions and knowledge relationships. If you are managing a system that serves web pages you may define a promise that port 80 needs to be open on a web server. This same web server may also define a promise that a particular directory has a particular set of permissions and the proper owner to serve web pages via Apache.
This simple but powerful idea allows a very practical uniformity in CFEngine syntax.
Promise Types
The promise_type
defines what kind of object is making the promise. The type
dictates how CFEngine interprets the promise body. These promise types are
straightforward: The files
promise type deals with file permissions and file
content, and the packages
promise type allows you to work with packaging
systems such as rpm and apt.
Some promise types are common to all CFEngine components, while others can
only be executed by one of them. cf-serverd
cannot keep packages
promises,
and cf-agent
cannot keep access
promises. See the
Promise Type reference for a comprehensive
list of promise types.
The Promiser
The promiser is an object affected by a promise, and this can be anything: a file, a port on a network. It is the entity that is making a promise that a certain fact will be true. These facts are listed in the form of attributes and values. A file could promise that a permission attribute has a particular value (i.e. 775 permission value) and that an owner attribute has another value (i.e. "root").
When a promise is made in CFEngine it is made to another entity - a promisee. A promisee is an optional part of a promise declaration. The promisee can help provide insight into the system's configuration, and may become relevant as your system grows in complexity.
The classes in a promise control the conditions that make the promise valid. Examples are the operating system on which the policy is executed, or the day of the week. More about that in the classes and decision making section.
Not all of these elements are necessary every time, but when you combine them they enable a wide range of behavior.
Promise Example
# Promise type
files:
"/home/mark/tmp/test_plain" -> "system blue team",
create => "true",
perms => owner("@(usernames)"),
comment => "Hello World";
In this example, the promise is about a file named test_plain
in the
directory /home/mark/tmp
, and the promise is made to some entity named
system blue team
. The create
attribute instructs CFEngine to create the
file if it doesn't exist. It has a list of owners that is defined by a
variable named "usernames" (see the documentation about
Bodies for more details on this last
expression).
The comment attribute in this example can be added to any promise. It has no actual function other than to provide more information to the user in error tracing and auditing.
This is a promise that will affect the state of a file on the filesystem. In
CFEngine you can do this without having to execute the touch
, chmod
, and
chown
commands. CFEngine is declarative: you declare a contract (or a
promise) that you want CFEngine to keep and you leave the details up to the
tool.
Promise Attributes
Promise attributes have a type and a value. The type can be any of the datatypes that are allowed for variables, and in addition
Boolean - allowed input values are
"true"
/"false"
"on"
/"off"
"yes"
/"no"
irange[min, max]
andrrange[min, max]
- a range of integer or real values, created via theirange()
andrrange()
functionsclist
- a list of classes or class expressions. Note that these attributes can take both strings (which are evaluated as class expressions) and functions that return typeclass
Menu option - one value from a list of values
body
type - a complex set of attributes expressed in a separate, reusable blockbundle
type - a separate bundle that is used as a sub-routine or a sub-set of promises
Implicit Promises
Some promise types can have implicit behavior. For example, the following promise simply prints out a log message "hello world".
reports:
"hello world";
The same promise could be implemented using the commands
type, invoking the
echo command:
commands:
"/bin/echo hello world";
These two promises have default attributes for everything except the `promiser'. Both promises simply cause CFEngine to print a message.
Bundles
A bundle is a collection of promises. They allow to group related promises together into named building blocks that can be thought of as "subroutines" in the CFEngine promise language. A bundle that groups a number of promises related to configuring a web server or a file system would be named "webserver" or "filesystem," respectively.
Most promise types are specific to a particular kind of interpretation that
requires a typed interpreter - the bundle type. Bundles belong to the agent
that is used to keep the promises in the bundle. So cf-agent
has bundles
declared as:
bundle agent my_name
{
}
while cf-serverd
has bundles declared as:
bundle server my_name
{
}
and cf-monitord
has bundles declared as
bundle monitor my_name
{
}
A number of promises can be made in any kind of bundle since they are of a
generic input/output nature. These are vars
, classes
, defaults
,
meta
and reports
promises.
Common Bundles
Bundles of type common
may only contain the promise types that are common to
all bodies. Their main function is to define cross-component global
definitions.
bundle common globals
{
vars:
"global_var" string = "value";
classes:
"global_class" expression = "value";
}
Common bundles are observed by every agent, whereas the agent specific bundle types are ignored by components other than the intended recipient.
Rules for evaluation of common bundles
These are the specific evaluation differences between common and agent bundles:
- common bundles are automatically evaluated even if they are not in the bundlesequence, as long as they have no parameters
- auto-evaluated common bundles (not in the bundlesequence explicitly) don't evaluate their
reports
promises, so their reports won't be printed. - when common bundles define a class, it's global (
scope
isnamespace
) by default; the classes in agent bundles are local (scope
isbundle
) by default. - common bundles can only contain
meta
,default
,vars
,classes
, andreports
promises
Bundle Parameters
Bundles can be parameterized, allowing for code re-use. If you need to do the same thing over and over again with slight variations, using a promise bundle is an easy way to avoid unnecessary duplication in your promises.
bundle agent hello_world
{
vars:
"myfiles" => "/tmp/world.txt";
"desired_content" string => "hello";
"userinfo" data => parsejson('{ "mark": 10, "jeang":20, "jonhenrik":30, "thomas":40, "eben":-1 }');
methods:
"Hello World"
usebundle => ensure_file_has_content("$(myfiles)", "$(desired_content)");
"report" usebundle => subtest_c(@(userinfo));
}
bundle agent ensure_file_has_content(file, content)
{
files:
"$(file)"
handle => "$(this.bundle)_file_content",
create => "true",
edit_defaults => empty,
edit_line => append_if_no_line("$(content)"),
comment => "Ensure that the given parameter for file '$(file)' has only
the contents of the given parameter for content '$(content)'";
}
bundle agent subtest_c(info)
{
reports:
"user ID of mark is $(info[mark])";
}
You can pass slist
and data
variables to other bundles with
the @(var)
notation. You do NOT need to qualify the variable name
with the current bundle name.
Scope
All variables in CFEngine are globally accessible. If you refer to a variable by ‘$(unqualified)’, then it is assumed to belong to the current bundle. To access any other (scalar) variable, you must qualify the name, using the name of the bundle in which it is defined:
$(bundle_name.qualified)
The value of the variable depends on evaluation order, which is not controllable by the user. Thus you should not assume that you can evaluate a bundle twice with different variables and get variables from it that correspond to the second evaluation. In other words, if you have:
bundle agent mybundle(x)
{
vars:
"y" string => $(x);
}
and call mybundle(1)
and mybundle(2)
, the variable y
could be 1
or 2
.
Classes defined inside agent
bundles are not visible outside
those bundles. Classes defined in common
bundles
have global scope, so they are visible everywhere.
Note that namespaced bundles work exactly the same way as
non-namespaced bundles (which are actually in the default
namespace). You just say namespace:bundle_name
instead of
bundle_name
. See Namespaces for more details.
Bodies
While the idea of a promise is very simple, the definition of a promise can
grow complicated. Complex promises are best understood by breaking them down
into independent, re-usable components. The CFEngine reserved word body
is
used to encapsulate the details of complex promise attribute values. Bodies
can optionally have parameters.
bundle agent example
{
files:
!windows::
"/etc/passwd"
handle => "example_files_not_windows_passwd",
perms => system;
"/home/bill/id_rsa.pub"
handle => "example_files_not_windows_bills_priv_ssh_key",
perms => mog("600", "bill", "sysop"),
create => "true";
}
The promisers in this example are the files /etc/passwd
and
/home/bill/id_rsa.pub
. The promise is that the perms
attribute type is
associated with a named, user-defined promise body system
and mog
respectively.
body perms system
{
mode => "644";
owners => { "root" };
groups => { "root" };
}
body perms mog(mode,user,group)
{
owners => { "$(user)" };
groups => { "$(group)" };
mode => "$(mode)";
}
Like bundles, bodies have a type. The type of the body has to match the left-hand side of the promise attribute in which it is used. In this case, files
promises have an attribute perms
that can be associated with any body of type perms
.
The attributes within the body are then type specific. Bodies of type perms
consist of the file permissions, the file owner, and the file group, which the instance system
sets to 644
, root
and root
, respectively.
Such bodies can be reused in multiple promises. Like bundles, bodies can have parameters. The body mog
also consists of the file permissions, file owner, and file group, but the values of those attributes are passed in as parameters.
Implicit, Control Bodies
A special case for bodies are the implicit promises that configure the basic
operation of CFEngine. These are hard-coded to CFEngine and control the basic
operation of the agents, such as cf-agent
and cf-serverd
. Each agent has a
special body whose name is control
.
body agent control
{
bundlesequence => { "test" };
}
This promise bodies configures the bundlesequence
to execute on a cf-agent.
body server control
{
allowconnects => { "127.0.0.1" , "::1", @(def.acl) };
}
This promise body defines the clients allowed to connect to a cf-serverd. For more information, see the reference documentation about the CFEngine Agents
Classes and Decisions
Classes are used to apply promises only to particular environments, depending on context. A promise might only apply to Linux systems, or should only be applied on Sundays, or only when a variable has a certain value.
Classes are simply facts that represent the current state or context of a system. The list of set classes classifies the environment at time of execution.
Classes are either set
or not set
, depending on context. Classes fall into
hard classes that are discovered by CFEngine, and soft classes that are
user-defined. Refer to Hard and Soft Classes in the Reference
section for more information.
In CFEngine Enterprise, the list of set classes is reported to the CFEngine Database Server and can be used there for reporting, grouping of hosts and inventory management.
Hard Classes
Hard classes are discovered by CFEngine. Each time it wakes up, it discovers and reads properties of the environment or context in which it runs.It turns these properties of the environment into classes. This information is effectively cached and may be used to make decisions about configuration.
You can see all of the classes defined on a particular host by running the following command as a privileged user.
$ cf-promises --show-classes|grep hardclass
These are classes that describe your operating system, the time of
day, the week of the year, etc. Time-varying classes (tagged with
time_based
) will change if you do this a few times over the course
of a week.
Soft Classes
Soft classes are user-defined classes which you can use to implement your own classifications. These classes are defined in bundles and are evaluated when the bundle is evaluated. They can be based on test functions or on other classes.
bundle agent myclasses
{
classes:
"solinux" expression => "linux||solaris";
"alt_class" or => { "linux", "solaris", fileexists("/etc/fstab") };
"oth_class" and => { fileexists("/etc/shadow"), fileexists("/etc/passwd") };
reports:
alt_class::
# This will only report "Boo!" on linux, solaris, or any system
# on which the file /etc/fstab exists
"Boo!";
}
This example defines a few soft classes local to the myclasses
bundle.
The
solinux
soft class is defined as a combination of thelinux
or thesolaris
hard classes. This class will be set if the operating system family is either of these values.The
alt_class
soft class is defined as a combination oflinux
,solaris
, or the presence of a file named/etc/fstab
. If one of the two hard classes evaluate to true, or if there is a file named/etc/fstab
, thealt_class
class will also be set.The
oth_class
soft class is defined as the combination of twofileexists
functions -/etc/shadow
and/etc/passwd
. If both of these files are present theoth_class
class will also be set.
Negative Knowledge
If a class is set, then it is certain that the corresponding fact is true. However, that a class is not set could mean that something is not the case, or that something is simply not known. This is only a problem with soft classes, where the state of a class can change during the execution of a policy, depending on the order in which bundles and promises are evaluated.
Making Decisions based on classes
The easiest way to limit the application of a promise to certain conditions is to use the following notation:
bundle agent greetings
{
reports:
Morning::
"Good morning!";
Evening::
"Good evening!";
}
In this example, the report "Good morning!" is only printed if the class
Morning
is set, while the report "Good evening!" is only printed when the
class Evening
is set.
A limitation of this notation is that the class expression needs to be
constant. The class predicate ifvarclass
can be used if variable class
expressions are required. It is AND
ed with the normal class expression, and
is evaluated together with the promise. It may contain variables as long as
the resulting expansion is a legal class expression.
bundle agent example
{
vars:
"french_cities" slist => { "toulouse", "paris" };
"german_cities" slist => { "berlin" };
"italian_cities" slist => { "milan" };
"usa_cities" slist => { "lawrence" };
"all_cities" slist => { @(french_cities), @(german_cities), @(italian_cities), @(usa_cities) };
classes:
"italy" or => { @(italian_cities) };
"germany" or => { @(german_cities) };
"france" or => { @(french_cities) };
reports:
"It's $(sys.date) here";
Morning.italy::
"Good morning from Italy",
ifvarclass => "$(all_cities)";
Afternoon.germany::
"Good afternoon from Germany",
ifvarclass => "$(all_cities)";
france::
"Hello from France",
ifvarclass => "$(all_cities)";
"Hello from $(all_cities)",
ifvarclass => "$(all_cities)";
}
Example Output:
cf-agent -Kf example.cf -D lawrence -b example
R: It's Tue May 28 16:47:33 2013 here
R: Hello from lawrence
cf-agent -Kf example.cf -D paris -b example
R: It's Tue May 28 16:48:18 2013 here
R: Hello from France
R: Hello from paris
cf-agent -Kf example.cf -D milan -b example
R: It's Tue May 28 16:48:40 2013 here
R: Hello from milan
cf-agent -Kf example.cf -D germany -b example
R: It's Tue May 28 16:49:01 2013 here
cf-agent -Kf example.cf -D berlin -b example
R: It's Tue May 28 16:51:53 2013 here
R: Good afternoon from Germany
R: Hello from berlin
In this example, lists of cities are defined in the vars
section and these
lists are combined into a list of all cities. These variable lists are used to
qualify the greetings and to make the policy more concise. In the classes
section a country class is defined if a class described on the right hand side
evaluates to true. In the reports section the current time is always reported
but only agents found to have the Morning
and italy
classes defined will
report "Good morning from Italy", this is further qualified by ensuring that
the report is only generated if one of the known cities also has a class
defined.
Operators and Precedence
Classes promises define new classes based on combinations of old ones. This is how to make complex decisions in CFEngine, with readable results. It is like defining aliases for class combinations. Such class 'aliases' may be specified in any kind of bundle.
Note that whitespace is not allowed between operators, so something like
a . b
, though perhaps more readable, will be rejected by the CFEngine parser.
Classes may be combined with the operators listed here in order from highest to lowest precedence:
‘()':: ~ The parenthesis group operator.
‘!’:: ~ The NOT operator.
‘.’:: ~ The AND operator.
‘&’:: ~ The AND operator (alternative).
‘|’:: ~ The OR operator.
‘||’:: ~ The OR operator (alternative).
These operators can be combined to form complex expressions. For example, the following expression would be only true on Mondays or Wednesdays from 2:00pm to 2:59pm on Windows XP systems:
(Monday|Wednesday).Hr14.WinXP::
Operands that are functions
If an operand is another function and the return value of the function is undefined, the result of the logical operation will also be undefined. For this reason, when using functions as operators, it is safer to collapse the functions down to scalar values and to test if the values are either true or false before using them as operands in a logical expression.
e.g.
...
classes:
"variable_1"
expression => fileexists("/etc/aliases.db");
...
"result"
or => { isnewerthan("/etc/aliases", "/etc/aliases.db"),
"!variable_1" };
The function, isnewerthan
can return "undefined" if one or other of the files
does not exist. In that case, result would also be undefined. By checking the
validity of the return value before using it as an operand in a logical expression,
unpredictable results are avoided. i.e negative knowledge does not necessarily
imply that something is not the case, it could simply be unknown. Checking if
each file exists before calling isnewerthan
would avoid this problem.
Global and Local classes
Classes defined in bundles of type common
are global in scope, whereas
classes defined in all other bundle types are local. Classes are evaluated
when the bundle is evaluated (and the bundles are evaluated in the order
specified in the bundlesequence
).
Note that any class promise must have one - and only one - value constraint. That is, you might not leave 'expression' in the example above or add both 'and' and 'xor' constraints to the single promise.
Another type of class definition uses the
body classes
. This allows setting of
classes based on the outcome of a promise. To set a class if a promise is
repaired, one might write:
"promiser..."
...
classes => if_repaired("signal_class");
These classes are global in scope, but the
scope
attribute can be used to make
them local to the bundle.
Finally, restart_class
classes in processes
are global.
Canceling classes
You can cancel a class with a classes
body.
See the cancel_kept
, cancel_notkept
, and cancel_repaired
attributes.
Class Scopes: A More Complex Example
body common control
{
bundlesequence => { "global","local_one", "local_two" };
}
#################################
bundle common global
{
classes:
# The soft class "zero" is always satisfied,
# and is global in scope
"zero" expression => "any";
}
#################################
bundle agent local_one
{
classes:
# The soft class "one" is always satisfied,
# and is local in scope to local_one
"one" expression => "any";
}
#################################
bundle agent local_two
{
classes:
# The soft class "two" is always satisfied,
# and is local in scope to ls_2
"two" expression => "any";
reports:
zero.!one.two::
# This report will be generated
"Success";
}
In this example, there are three bundles. One common bundle named global
with a global scope. Two agent bundles define classes one
and two
which
are local to those bundles.
The local_two
bundle promises a report "Success" which applies only if
zero.!one.two
evaluates to true. Within the local_two
scope this evaluates
to true
because the one
class is not set.
Controlling Frequency
When checking a series of expensive functions and verifying complex promises, you may want to make sure that CFEngine is not checking too frequently. One way of doing this is classes and class expression, another is using locks.
CFEngine incorporates a series of locks which prevent it from checking promises too often, and which prevent it from spending too long trying to check promises it has recently verified. This locking mechanism works in such a way that you can start several CFEngine components simultaneously without them interfering with each other. You can control two things about each kind of action in CFEngine:
ifelapsed
The minimum time (in minutes) which should have passed since the last time that promise was verified. It will not be executed again until this amount of time has elapsed. Default time is 1 minute.
expireafter
The maximum amount (in minutes) of time cf-agent
should wait for an old
instantiation to finish before killing it and starting again. You can think
about expireafter
as a timeout to use when a promise verification may
involve an operation that could wait indefinitely. Default time is 120
minutes.
You can set these values either globally (for all actions) or for each action
separately. If you set global and local values, the local values override the
global ones. All times are written in units of minutes. The following global
setting is defined in body agent control
.
body agent control
{
ifelapsed => "60"; # one hour
}
This setting tells CFEngine not to verify promises until 60 minutes have
elapsed, ie ensures that the global frequency for all promise verification is
one hour. This global setting of one hour could be changed for a specific
promise body by setting ifelapsed
in the promise body.
body action example
{
ifelapsed => "90"; # 1.5 hours
}
This promise which overrides the global 60 minute time period and defines a frequency of 90 minutes.
These locks do not prevent the whole of cf-agent
from running, only
atomic promise checks on the same objects (packages, users, files,
etc.). Several different cf-agent
instances can run concurrently.
The locks ensure that promises will not be verified by two cf-agents
at the same time or too soon after a verification.
Variables
Just like classes are defined as promises, variables (or "variable definitions") are also promises. Variables can be defined in any promise bundle. This bundle name can be used as a context when using variables outside of the bundle they are defined in.
CFEngine variables have three high-level types: scalars, lists, and data containers.
- A scalar is a single value,
- a list is a collection of scalars.
- a data container is a lot like a JSON document, it can be a key-value map or an array or anything else allowed by the JSON standard with unlimited nesting.
Scalar Variables
Each scalar may have one of three types: string, int or real. String scalars are sequences of characters, integers are whole numbers, and reals are float pointing numbers.
vars:
"my_scalar" string => "String contents...";
"my_int" int => "1234";
"my_real" real => "567.89";
Integer constants may use suffixes to represent large numbers. The following suffixes can be used to create integer values for common powers of 1000.
- 'k' = value times 1000
- 'm' = value times 10002
- 'g' = value times 10003
Since computing systems such as storage and memory are based on binary values, CFEngine also provide the following uppercase suffixes to create integer values for common powers of 1024.
- 'K' = value times 1024.
- 'M' = value times 10242
- 'G' = value times 10243
However, the values must have an integer numeric part (e.g. 1.5M is not allowed).
In some contexts, %
can be used a special suffix to denote percentages.
Lastly, there is a reserved value which can be used to specific a parameter as having no limit at all.
- 'inf' = a constant representing an unlimited value.
CFEngine typing is mostly dynamic, and CFEngine will try to coerce string values into int and real types, and if it cannot it will report an error. However, arguments to built-in functions check the defined argument type for consistency.
Scalar Referencing and Expansion
Scalar variables are referenced by $(my_scalar)
(or ${my_scalar}
) and
expand to the single value they hold at that time. If you refer to a variable
by $(unqualified)
, then it is assumed to belong to the current bundle. To
access any other (scalar) variable, you must qualify the name, using the name
of the bundle in which it is defined:
$(bundle_name.qualified)
Scalar Size Limitations
At the moment, up to 4095 bytes can fit into a scalar variable. This limitation may be removed in the future.
If you try to expand strings in a variable or string context that add
up to more that 4095 bytes, you will notice this limitation as well.
The functions eval()
to do math, string_head()
and string_tail()
to extract a certain number of characters from either end of a string,
and string_length()
to find a string's length may be helpful.
See readfile()
for more detail on reading values from a file.
See data_readstringarray()
and data_readstringarrayidx()
for a way
to read large files' contents into a data container without going
through scalar variables or arrays.
Lists
List variables can be of type slist
, ilist
or rlist
to hold lists of
strings, integers or reals, respectively.
Every element of a list is subject to the same size limitations as a regular scalar.
They are declared as follows:
vars:
"my_slist" slist => { "list", "of", "strings" };
"my_ilist" ilist => { "1234", "5678" };
"my_rlist" rlist => { "567.89" };
List Substitution and Expansion
An entire list is referenced with the symbol ‘@’ and can be passed in their
entirety in any context where a list is expected as @(list)
. For example,
the following variable definition references a list named "shortlist":
vars:
"shortlist" slist => { "you", "me" };
"longlist" slist => { @(shortlist), "plus", "plus" };
The declaration order does not matter – CFEngine will understand the
dependency, and execute the promise to assign the variable @(shortlist)
before the promise to assign the variable @(longlist)
.
Using the @ symbol in a string scalar will not result in list substitution. For example, the string value "My list is @(mylist)" will not expand this reference.
Using the scalar reference to a local list variable, will cause CFEngine to
iterate over the values in the list. E.g. suppose we have local list variable
@(list)
, then the scalar $(list)
implies an iteration over every value of
the list.
In some function calls, listname
instead of @(listname)
is
expected. See the specific function's documentation to be sure.
Data Container Variables
The data
containers can contain several levels of data structures,
e.g. list of lists of key-value arrays. They are used to store
structured data, such as data read from JSON files. The variable type
is data
.
Data containers are obtained from functions that return data
types,
such as readjson()
or parsejson()
, or from merging existing
containers.
They can NOT be modified, once created.
Data containers do not have the size limitations of regular scalar variables.
Associative Arrays
Note that associative arrays are being deprecated in favor of the data
variable type. It is recommended to use the data
variable type instead
whenever possible to ensure future compatibility of your CFEngine policy.
Every value in an associative array is subject to the same size limitations as a regular scalar.
Associative array variables are written with [
and ]
brackets that enclose
an arbitrary key. These keys are associated with values
bundle agent example
{
vars:
"component" slist => { "cf-monitord", "cf-serverd", "cf-execd" };
"array[cf-monitord]" string => "The monitor";
"array[cf-serverd]" string => "The server";
"array[cf-execd]" string => "The executor, not executioner";
commands:
"/bin/echo $(component) is"
args => "$(array[$(component)])";
}
This example defines three values in an associative array under the keys
cf-monitord
, cf-serverd
, and cf-execd
. They and are sequently printed
with the echo command.
Arrays are associative and may be of type scalar or list. Enumerated arrays are simply treated as a special case of associative arrays, since there are no numerical loops in CFEngine. Special functions exist to extract lists of keys from array variables for iteration purposes.
Here is an example of using the function getindices()
which
extracts all of the keys from an associative array. If this series of promises
were executed it would print out two messages, one for each key.
bundle agent array
{
vars:
"v[index_1]" string => "value_1";
"v[index_2]" string => "value_2";
"parameter_name" slist => getindices("v");
reports:
"Found index: $(parameter_name)";
}
Normal Ordering
CFEngine takes a pragmatic point of view to ordering. When promising scalar
attributes and properties, ordering is irrelevant and should not be considered.
More complex patterned data structures require ordering to be preserved, e.g.
editing in files. CFEngine solves this in a two-part strategy:
CFEngine maintains a default order of promise-types. This is based on a simple
logic of what needs to come first, e.g. it makes no sense to create something
and then delete it, but it could make sense to delete and then create (an
equilibrium). This is called normal ordering and is described below.
You can override normal ordering in exceptional circumstances by making a
promise in a class context and defining that class based on the outcome of
another promise, or using the depends_on
promise attribute.
Agent normal ordering
CFEngine tries to keep variable and class promises before starting to consider any other kind of promise. In this way, global variable and classes can be set.
If you set variables based on classes that are determined by other variables, then you introduce an order dependence to the resolution that might be non-unique. Since CFEngine starts trying to converge values as soon as possible, it is best to define variables in bundles before using them, i.e. as early as possible in your configuration. In general it is wise to avoid class-variable dependency as much as possible.
CFEngine executes agent promise bundles in the strict order defined by the
bundlesequence
(possibly overridden by the -b
or --bundlesequence
command line option).
Within a bundle, the promise types are executed in a round-robin fashion
according to so-called normal ordering
(essentially deletion first, followed
by creation). The actual sequence continues for up to three iterations of the
following, converging towards a final state:
meta
vars
defaults
classes
users
files
packages
guest_environments
methods
processes
services
commands
storage
databases
reports
Within edit_line
bundles in files promises,
the normal ordering is:
meta
vars
defaults
classes
delete_lines
field_edits
insert_lines
replace_patterns
reports
The order of promises within one of the above types follows their top-down
ordering within the bundle itself. The order may be overridden by making a
promise depend on a class that is set by another promise, or by using the
depends_on
attribute in the promise.
Server normal ordering
As with the agent, common bundles are executed before any server bundles;
following this all server bundles are executed (the bundlesequence
is only
used for cf-agent). Within a server bundle, the promise types are unambiguous.
Variables and classes are resolved in the same way as the agent. On
connection, access control must be handled first, then a role request might be
made once access has been granted. Thus ordering is fully constrained by
process with no additional freedoms.
Within a server bundle, the normal ordering is:
vars
classes
access
roles
Monitor normal ordering
As with the agent, common bundles are executed before any monitor bundles;
following this all monitor bundles are executed (the bundlesequence
is only
used for cf-agent). Variables and classes are resolved in the same way as the
agent.
Within a monitor bundle, the normal ordering is:
vars
classes
measurements
reports
Loops
There are no explicit loops in CFEngine, instead there are lists. To make a loop, you simply refer to a list as a scalar and CFEngine will assume a loop over all items in the list.
It's as if you said "I know three colors: red green blue. Let's talk about color."
body common control
{
bundlesequence => { "color_example" };
}
bundle agent color_example
{
vars:
"color" slist => { "red", "green", "blue" };
reports:
"Let's talk about $(color)";
}
CFEngine will implicitly loop over each $(color)
:
% cf-agent -K -f ./test_colors.cf
R: Let's talk about red
R: Let's talk about green
R: Let's talk about blue
Here's a more complex example.
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
vars:
"component" slist => { "cf-monitord", "cf-serverd", "cf-execd" };
"array[cf-monitord]" string => "The monitor";
"array[cf-serverd]" string => "The server";
"array[cf-execd]" string => "The executor, not executionist";
reports:
"$(component) is $(array[$(component)])";
}
In this example, the list component
has three elements. The list as a whole
may be referred to as @(component)
, in order to pass the whole list to a
promise where a list is expected. However, if we write $(component)
,
i.e. the scalar variable, then CFEngine will substitute each scalar from the
list in turn, and thus iterate over the list elements using a loop.
The output looks something like this:
$ cf-agent unit_loops.cf
2013-06-12T18:56:01+0200 notice: R: cf-monitord is The monitor
2013-06-12T18:56:01+0200 notice: R: cf-serverd is The server
2013-06-12T18:56:01+0200 notice: R: cf-execd is The executor, not executionist
You see from this that, if we refer to a list variable using the scalar
reference operator $()
, CFEngine interprets this to mean “please iterate
over all values of the list”. Thus, we have effectively a `foreach' loop,
without the attendant syntax.
If a variable is repeated, its value is tied throughout the expression; so the output of:
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
vars:
"component" slist => { "cf-monitord", "cf-serverd", "cf-execd" };
"array[cf-monitord]" string => "The monitor";
"array[cf-serverd]" string => "The server";
"array[cf-execd]" string => "The executor, not executioner";
commands:
"/bin/echo $(component) is"
args => "$(array[$(component)])";
}
is as follows:
2013-06-12T18:57:34+0200 notice: Q: ".../bin/echo cf-mo": cf-monitord is The monitor
2013-06-12T18:57:34+0200 notice: Q: ".../bin/echo cf-se": cf-serverd is The server
2013-06-12T18:57:34+0200 notice: Q: ".../bin/echo cf-ex": cf-execd is The executor, not executioner
Iterating Across Multiple Lists
CFEngine can iterate across multiple lists simultaneously.
bundle agent iteration
{
vars:
"stats" slist => { "value", "av", "dev" };
"monvars" slist => {
"rootprocs",
"otherprocs",
"diskfree",
"loadavg"
};
reports:
"mon.$(stats)_$(monvars) is $(mon.$(stats)_$(monvars))";
}
This example uses two lists, stats
and monvars
. We can now iterate over both lists in the same promise. The reports that we thus generate will report on value_rootprocs
, av_rootprocs
, and dev_rootprocs
, followed next by value_otherprocs
, av_otherprocs
, etc, ending finally with dev_loadavg
.
The order of iteration is an implementation detail and should not be expected to be consistent. Use the sort()
function if you need to sort a list in a predictable way.
Pattern Matching and Referencing
One of the strengths of CFEngine 3 is the ability to recognize and exploit patterns. All string patterns in CFEngine 3 are matched using PCRE regular expressions.
CFEngine has the ability to extract back-references from pattern matches. This makes sense in two cases. Back references are fragments of a string that match parenthetic expressions. For instance, suppose we have the string:
Mary had a little lamb ...
and apply the regular expression
"Mary ([^l]+)little (.*)"
The pattern matches the entire string, and it contains two parenthesized
subexpressions, which respectively match the fragments had a
and lamb
...
. The regular expression libraries assign three matches to this result,
labelled 0, 1 and 2.
The zeroth value is the entire string matched by the total expression. The first value is the fragment matched by the first parenthesis, and so on.
Each time CFEngine matches a string, these values are assigned to a special
variable context $(match.n)
. The fragments can be referred to in the remainder
of the promise. There are two places where this makes sense. One is in pattern
replacement during file editing, and the other is in searching for files.
Consider the examples below:
bundle agent testbundle
{
files:
# This might be a dangerous pattern - see explanation in the next section
# on "Runaway change warning"
"/home/mark/tmp/cf([23])?_(.*)"
edit_line => myedit("second backref: $(match.2)");
}
There are other filenames that could match this pattern, but if, for example,
there were to exist a file /home/mark/tmp/cf3_test
, then we would have:
‘$(match.0)’
equal to `/home/mark/tmp/cf3_test'
‘$(match.1)’
equal to `3'
‘$(match.2)’
equal to `test'
Note that because the pattern allows for an optional '2' or '3' to follow the
letters cf
, it is possible that $(match.1)
would contain the empty string.
For example, if there was a file named /home/mark/tmp/cf_widgets
, then we
would have
‘$(match.0)’
equal to `/home/mark/tmp/cf_widgets'
‘$(match.1)’
equal to `'
‘$(match.2)’
equal to `widgets'
Now look at the edit bundle. This takes a parameter (which is the
back-reference from the filename match), but it also uses back references to
replace shell comment lines with C comment lines (the same approach is used to
hash-comment lines in files). The back-reference variables $(match.n)
refer
to the most recent pattern match, and so in the C_comment
body, they do not
refer to the filename components, but instead to the hash-commented line in
the replace_patterns
promise.
bundle edit_line myedit(parameter)
{
vars:
"edit_variable" string => "private edit variable is $(parameter)";
insert_lines:
"$(edit_variable)";
replace_patterns:
# replace shell comments with C comments
"#(.*)"
replace_with => C_comment,
select_region => MySection("New section");
}
########################################
# Bodies
########################################
body replace_with C_comment
{
replace_value => "/* $(match.1) */"; # backreference from replace_patterns
occurrences => "all"; # first, last, or all
}
########################################################
body select_region MySection(x)
{
select_start => "\[$(x)\]";
select_end => "\[.*\]";
}
Try this example on the file
[First section]
one
two
three
[New section]
four
#five
six
[final]
seven
eleven
The resulting file is edited like this:
[First section]
one
two
three
[New section]
four
/* five */
six
[final]
seven
eleven
private edit variable is second backref: test
Runaway change warning
Be careful when using patterns to search for files that are altered by
CFEngine if you are not using a file repository. Each time CFEngine makes a
change it saves an old file into a copy like cf3_test.cf-before-edit
. These
new files then get matched by the same expression above – because it ends in
the generic.*
), or does not specify a tail for the expression. Thus CFEngine
will happily edit backups of the edit file too, and generate a recursive
process, resulting in something like the following:
cf3_test cf3_test.cf-before-edit
cf3_test~ cf3_test~.cf-before-edit.cf-before-edit
cf3_test~.cf-before-edit cf3_test~.cf-before-edit.cf-before-edit.cf-before-edit
Always try to be as specific as possible when specifying patterns. A lazy approach will often come back to haunt you.
Commenting lines
The following example shows how you would hash-comment lines in a file using CFEngine.
######################################################################
#
# HashCommentLines implemented in CFEngine 3
#
######################################################################
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
}
########################################################
bundle agent testbundle
{
files:
"/home/mark/tmp/comment_test"
create => "true",
edit_line => comment_lines_matching;
}
########################################################
bundle edit_line comment_lines_matching
{
vars:
"regexes" slist => { "one.*", "two.*", "four.*" };
replace_patterns:
"^($(regexes))$"
replace_with => comment("# ");
}
########################################
# Bodies
########################################
body replace_with comment(c)
{
replace_value => "$(c) $(match.1)";
occurrences => "all";
}
Regular expressions in paths
When applying regular expressions in paths, the path will first be split at
the path separators, and each element matched independently. For example, this
makes it possible to write expressions like /home/.*/file
to match a single
file inside a lot of directories — the .*
does not eat the whole string.
Note that whenever regular expressions are used in paths, the /
is always
used as the path separator, even on Windows. However, on Windows, if the
pathname is interpreted literally (no regular expressions), then the backslash
is also recognized as the path separator. This is because the backslash has a
special (and potentially ambiguous) meaning in regular expressions (a \d
means the same as [0-9]
, but on Windows it could also be a path separator
and a directory named d
).
The pathtype
attribute allows you to force a specific behavior when
interpreting pathnames. By default, CFEngine looks at your pathname and makes
an educated guess as to whether your pathname contains a regular expression.
The values literal
and regex
explicitly force CFEngine to interpret the
pathname either one way or another. (see the pathtype
attribute).
body common control
{
bundlesequence => { "wintest" };
}
########################################
bundle agent wintest
{
files:
"c:/tmp/file/f.*" # "best guess" interpretation
delete => nodir;
"c:\tmp\file"
delete => nodir,
pathtype => "literal"; # force literal string interpretation
"C:/windows/tmp/f\d"
delete => nodir,
pathtype => "regex"; # force regular expression interpretation
}
########################################
body delete nodir
{
rmdirs => "false";
}
Note that the path /tmp/gar.*
will only match filenames like /tmp/gar
,
/tmp/garbage
and /tmp/garden
. It will not match filename like
/tmp/gar/baz
(because even though the .*
in a regular expression means
"zero or more of any character", CFEngine restricts that to mean "zero or more
of any character in a path component"). Correspondingly, CFEngine also
restricts where you can use the /
character (you can't use it in a character
class like [^/]
or in a parenthesized or repeated regular expression
component.
This means that regular expressions which include "optional directory
components" won't work. You can't have a files promise to tidy the directory
(/usr)?/tmp
. Instead, you need to be more verbose and specify
/usr/tmp|/tmp
, or even better, think declaratively and create an slist
that contains both the strings /tmp
and /usr/tmp
, and then allow CFEngine
to iterate over the list!
This also means that the path /tmp/.*/something
will match files like
/tmp/abc/something
or /tmp/xyzzy/something
. However, even though the
pattern .*
means "zero or more of any character (except /
)", CFEngine
matches files bounded by directory separators. So even though the pathname
/tmp//something
is technically the same as the pathname /tmp/something
,
the regular expression /tmp/.*/something
will not match on the degenerate
case of /tmp//something
(or /tmp/something
).
Anchored vs. unanchored regular expressions
CFEngine uses the full power of regular expressions, but there are two “flavors” of regex. Because they behave somewhat differently (while still utilizing the same syntax), it is important to know which one is used for a particular component of CFEngine:
An “anchored” regular expression will only successfully match an entire
string, from start to end. An anchored regular expression behaves as if it
starts with ^
and ends with $
, whether you specify them yourself or not.
Furthermore, an anchored regular expression cannot have these automatic
anchors removed.
An “unanchored” regular expression may successfully match anywhere in a
string. An unanchored regex may use anchors (such as ^
, $
, \A
, \Z
,
\b
, etc.) to restrict where in the string it may match. That is, an
unanchored regular expression may be easily converted into a partially- or
fully-anchored regex.
For example, the comment parameter in readstringarray()
is an unanchored regex. If you specify the regular expression as #.*
, then
on any line which contains a pound sign, everything from there until the end
of the line will be removed as a comment. However, if you specify the regular
expression as ^#.*
(note the ^
anchor at the start of the regex), then
only lines which start with a #
will be removed as a comment! If you want to
ignore C-style comment in a multi-line string, then you have to a bit more
clever, and use this regex: (?s)/\*.*?\*/
Conversely, delete_lines
promises use anchored regular expressions to delete
lines. If our promise uses bob:\d*
as a line-matching regex, then only the
second line of this file will be deleted (because only the second line starts
with bob:
and is then followed exclusively by digits, all the way to the end
of the string).
bobs:your:uncle
bob:111770
thingamabob:1234
robert:bob:xyz
i:am:not:bob
If CFEngine expects an unanchored regular expression, then finding every line
that contains the letters bob
is easy. You just use the regex bob
. But if
CFEngine expects an anchored regular expression, then you must use .*bob.*
.
If you want to find every line that has a field which is exactly bob
with no
characters before or after, then it is only a little more complicated if
CFEngine expects an unanchored regex: (^|:)bob(:|$)
. But if CFEngine expects
an anchored regular expression, then it starts getting ugly, and you'd need to
use bob:.*|.*:bob:.*|.*:bob
.
Special topics on Regular Expressions
Regular expressions are a complicated subject, and really are beyond the scope of this document. However, it is worth mentioning a couple of special topics that you might want to know of when using regular expressions.
The first is how to not get a back reference. If you want to have a
parenthesized expression that does not generate a back reference, there is a
special PCRE syntax to use. Instead of using ()
to bracket the piece of a
regular expression, use (?:)
instead. For example, this will match the
filenames foolish, foolishly, bearish, bearishly, garish, and garishly in the
/tmp
directory. The variable $match.0
will contain the full filename, and
$match.1
will either contain the string ly
or the empty string. But the
(?:expression)
which matches foo, bear, or gar does not create a
back-reference:
files:
"/tmp/(?:foo|bear|gar)ish(ly)?"
Note that sometimes multi-line strings are subject to be matched by regular
expressions. CFEngine internally matches all regular expressions using
PCRE_DOTALL option, so .
matches newlines. If you want to match any
character except newline you could use \N
escape sequence.
Another thing you might want to do is ignore capitalization. CFEngine is
case-sensitive (in all things), so the files promise /tmp/foolish
will not
match the files /tmp/Foolish
or /tmp/fOoLish
, etc. There are two ways to
achieve case-insensitivity. The first is to use character classes:
files:
"/tmp/[Ff][Oo][Oo][Ll][Ii][Ss][Hh]"
While this is certainly correct, it can also lead to unreadability. The PCRE patterns in CFEngine have another way of introducing case-insensitivity into a pattern:
files:
"/tmp/(?i:foolish)"
The (?i:)
brackets impose case-insensitive matching on the text that it
surrounds, without creating a sub-expression. You could also write the regular
expression like this (but be aware that the two expressions are different, and
work slightly differently, so check the documentation for the specifics):
files:
"/tmp/(?i)foolish"
The /s
, /m
, and /x
switches from PCRE are also available, but use them
with great care!
Namespaces
Namespaces are private bundle and body "playgrounds", allowing multiple files to define the bundles and bodies with the same name in different namespaces without conflict. They are key to writing reusable policies.
Everything in CFEngine lives in a namespace (it's the default
namespace if not set).
Specifying a namespace
To isolate a file into its own namespace, you add a control promise to the
file before the relevant bundles or bodies. All bundles and bodies start off in the
default
namespace if you don't explicitly set this. Once set, this applies
until the end of the file or the next namespace change.
body file control
{
namespace => "myspace";
}
Accessing syntax elements between namespaces and the default namespace
To distinguish the bundle mymethod
in the default namespace from one in
another namespace, you prefix the bundle name with the namespace, separated by
a colon.
methods:
"namespace demo" usebundle => myspace:mymethod("arg1");
"namespace demo" usebundle => mymethod("arg1","arg2");
To distinguish a body from one in another namespace, you can prefix the body name with the namespace, separated by a colon.
files:
"/file"
create => "true",
perms => name1:settings;
If you don't make any namespace declarations, you'll be in the
default
namespace. Bundles, bodies, classes, and variables from the
default
namespace can be accessed like any other:
files:
"/file"
create => "true",
perms => default:settings;
If you use the standard library from your own namespace, remember to
specify this default:
prefix.
To access classes, variables, or meta-data in bundles in a different namespace, use the colon as a namespace prefix:
$(namespace:bundle.variable)
$(namespace:bundle_meta.variable)
Note that this means that if you are in a namespace that's not default
, you must qualify classes from default
fully:
default:myclass::
"do something" ifvarclass => "default:myotherclass";
Namespacing of classes and variables created in policy
In policy, you can't create classes outside your own namespace. So
the following, for example, will create the class mynamespace:done
if it runs in the namespace mynamespace
.
files:
"/file"
create => "true",
action => if_repaired("done");
Similarly, variables you create in a namespaced bundle have to be
prefixed like mynamespace:mybundle.myvar
from outside your
namespace, but can use mybundle.myvar
inside the namespace and
myvar
inside mybundle
.
As a workaround, you could have a helper bundle in another namespace to create classes and variables as needed.
Exceptions to namespacing rules
Exceptions to the rules above:
All hard classes can be used as-is from any namespace, without a namespace prefix. These are classes like
linux
. They will have the taghardclass
.All special variable contexts, as documented in Special Variables, are always accessible without a namespace prefix. For example,
this
,mon
,sys
, andconst
fall in this category.
Design Center Overview
The Design Center is a public repository for data-driven policy templates called sketches that is deeply integrated with CFEngine Enterprise.
Sketches are ready-to-use components (e.g. collections of bundles, support files, etc.) that can be directly imported and used in CFEngine policies. Most sketches are specialized for achieving specific tasks, or for maintaining a specific piece of software, but their scope and capabilities can range beyond. They are organized in categories and tagged according to their functionality.
Sketches are managed (installed, configured, enabled and uninstalled)
through a specialized tool called cf-sketch
which in turn talks to a
specialized Design Center API. That API is available for third-party
integrations.
In CFEngine Enterprise, the Design Center interactions are driven by the Mission Portal Design Center App for great ease of use.
This is a guide to the Design Center functionality. For the reference pages for its API and code structure, start with Design Center.
- Use Design Center Sketches to Deploy Policy
- The Design Center Sketch Workflow
- Value of Sketches to Users
- Design Center Terminology
- See Also
Use Design Center Sketches to Deploy Policy
The CFEngine Design Center is a collection of data-driven policy templates called sketches. Sketches can be configured and deployed ("activated") without prior knowledge of the CFEngine language.
The Design Center also contains tools that help you to manipulate and manage sketches:
Enterprise users manage and deploy sketches in the Mission Portal Design Center App. Deploy your first policy in the Design Center.
Community users manage and deploy sketches on the command line by using the
cf-sketch
tool.third-part integrators can use the Design Center API from the command line or through the CFEngine Enterprise API. For more information, see The Design Center API. <!-- (??? LINK TO Enterprise DC API depends on https://dev.cfengine.com/issues/6011 ???) -->
The Design Center Sketch Workflow
A sketch is installed, configured, and deployed (the whole process is called an "activation"), as shown in the diagram below:
Install a sketch from a master repository. At this point, the sketch is merely a template that cannot do anything because it doesn’t contain parameters. With CFEngine Enterprise, installations are handled invisibly.
Configure the sketch by providing parameters. Now, you can create sketch configurations. One sketch can have multiple configurations with different parameter sets that will get applied under different conditions. With CFEngine Enterprise, you do this from a GUI screen.
Deploy the runfile. This makes the sketch visible to CFEngine, so it can begin to execute. Typically the intermediate stage here is to generate a runfile (the policy glue that enables all the Design Center magic) and check it into your version control repository. With CFEngine Enterprise, this step is handled invisibly, you just enter a commit message for the change you've made.
Value of Sketches to Users
Sketches provide value to CFEngine users, especially Enterprise customers:
Instead of writing a single policy for each desired state, you can use data-driven policies (sketches) to implement, activate, or enforce policy. Users can configure the sketch for the appropriate environment and then deploy it to specific hosts or groups of hosts without knowing the intricacies of CFEngine policy language.
You can have infinitely flexible configurations of sketches across your infrastructure based on static (such as CPU or OS) or dynamic (such as time of day or number of users) conditions (classes). Every time CFEngine runs, it chooses which sketch(es) to execute with which parameter sets based on those conditions. As conditions change, the execution of sketches can change, just as the behavior of regular CFEngine policy can change.
There's a large list of sketches you can use already and the list is growing. You can use them as they are, or learn from their techniques and write your own.
Design Center Terminology
Design Center
Refers to the collection of sketches and the tools that allow you to manipulate and manage them.Design Center app (UI)
Refers to the Design Center user interface app that is located on the Mission Portal console for CFEngine Enterprise users.Design Center in GitHub
Refers to the CFEngine github repository of sketches, tools, and policy examples.Design Center API
Refers to the API which performs all operations related to sketches, parameter sets, environments, validations, and deployment.
See Also
- Deploy your first Policy (Enterprise) This Enterprise-specific tutorial illustrates how to use a sketch to configure and deploy a simple policy by using the Design Center App on the Mission Portal console.
- Configure Sketches on the Command Line (Community) This section is for Community users who plan to install, configure, and deploy sketches.
- Write a new Sketch (Enterprise and Community) This is an advanced section for both Enterprise and Community users who plan to write new sketches.
Command Line Sketches
The CFEngine Design Center is a repository of pre-made components
called sketches that allow you to use the full power of CFEngine
without having to learn the CFEngine policy language. Although
sketches are themselves written in the CFEngine policy language, you
can make use of them by simply installing them, configuring them using
the appropriate parameters, and deploying them on your
infrastructure. For users of the Community edition of CFEngine, this can be easily done
using both a command-line interface called cf-sketch
.
Please note that CFEngine Enterprise comes with the Design Center App to make the work below effortless.
Overview
This page provides instructions on how a Design Center sketch can be found, installed, configured, and executed as policy, using command-line tools with CFEngine community. The overview is as follows:
Before you Begin
Instructions
Step 1. Check out the Design Center repository
Step 5. Activate a sketch Define the parameter set and environment, and run the activate command.
Step 6. Deploy the sketch Generate and execute the runfile.
Additional resources are included at the end of this page.
Before you Begin
Complete the software requirements and review the basic concepts of sketches before you begin the instructions:
Requirements
To follow these instructions, you will need the following:
- A Unix-like system
- The following programs installed (all of these are either included or easily available in the package repositories for most operating system):
- CFEngine Community 3.5.0 or newer, or CFEngine Enterprise 3.5.0 or newer. Many of the components will work with CFEngine Community 3.4.0 or later (Enterprise 3.0.0 or later), but you need 3.5.0 to have access to the full range of features and sketches.
- Optional but recommended: the
Term::ReadLine::Gnu
Perl module. On many systems it is available in the standard package repositories (for example, on Ubuntu Linux you can install it usingapt-get install libterm-readline-gnu-perl
). You can also install it using thecpan
utility included with Perl.
Basic Concepts
The following concepts provide a better understanding of how the Design Center works. The Design Center framework contains the following:
- Sketches contain the code that is executed by CFEngine to
perform some task. Sketches are contributed by the CFEngine
community and hosted in the Design Center repositories. Most
sketches take a few parameters to configure their precise
behavior. For example, the
System::tzconfig
sketch contains the code to update the appropriate files in the system to set the correct timezone. - Parameter sets contain the parameters that tell a CFEngine sketch
the details of what to do. For example, the
System::tzconfig
sketch takes a parameter that tells it which timezone should be set in the system. The Design Center framework can store several parameter sets (each one identified by a name) for the same sketch, and from which you can choose to apply according to arbitrary circumstances. - Environments contain conditions, expressed as CFEngine Class Expressions, that indicate when and where a particular sketch will be executed with a particular set of parameters. Environments also contain expressions that determine when and where test and verbose modes will be enabled for a sketch. The Design Center can also store multiple named environment definitions, which you can combine in arbitrary ways to fine tune the execution of each sketch on each machine.
Sketches, Parameters and Environments, by themselves, do nothing. They have to be combined into Activations:
An activation is a combination of a sketch, a parameter set, and an
environment (all three specified by name) that defines which sketches
must be executed under which conditions, and with which parameters. It is
only through activations that you put sketches to work and make the
decisions as to how different parts of your infrastructure will be
configured. For example, you can have two different parameter sets
for the System::tzconfig
sketch, one for Linux and another one for
Solaris machines. You can then, on the central CFEngine Policy
Server, create two activations:
- Activation #1: Sketch
System::tzconfig
, parameter settzconfig-linux
, Environmentlinux
- Activation #2: Sketch
System::tzconfig
, parameter settzconfig-solaris
, Environmentsolaris
When these activations are distributed by the policy server to all the
clients, only Linux and Solaris machines will execute the
System::tzconfig
sketch, and each one of them will apply the
appropriate parameters.
Other concepts exist that can make your use of the Design Center even more powerful, but these are enough to get you started and to be able to follow these instructions.
Instructions
Step 1. Check out the Design Center repository
The Design Center is an open source project and is hosted on GitHub. Access the repository at https://github.com/cfengine/design-center.
The best way to get the Design Center at the moment is to check out
its git repository. For these instructions, we check it out
under the $HOME/source/
directory (you can use any location you
want, just replace it throughout the following instructions):
mkdir $HOME/source
cd $HOME/source
git clone https://github.com/cfengine/design-center.git
A directory called design-center
is created. We
call this the CHECKOUT directory and henceforth refer to it as
$CHECKOUT
. In the examples that follow, CHECKOUT is
$HOME/source/design-center
. You can save some typing by typing
export CHECKOUT=$HOME/source/design-center
at the prompt. From that point on, all command-line interaction can
use $CHECKOUT
and it will expand to the installation directory.
Step 2. Run cf-sketch
You can run cf-sketch in interactive mode directly from its directory
under $CHECKOUT
:
cd $CHECKOUT/tools/cf-sketch
./cf-sketch.pl
Welcome to cf-sketch version 3.5.0b1.
CFEngine AS, 2013.
Enter any command to cf-sketch, use 'help' for help, or 'quit' or '^D' to quit.
cf-sketch> _
By default, cf-sketch
provides an interactive prompt where you can
type the commands you want to execute. Type help
at the prompt
to see the descriptions of all the available commands. You can run
commands non-interactively by passing them as arguments to the
cf-sketch.pl
script from the command line.
Step 3. Search for sketches
The first step is to find some sketches to install on your system. The search command provides, without any parameters, a list of all available sketches:
cf-sketch> search
The following sketches are available:
Applications::Memcached Sketch for installing, configuring, and starting memcached.
...
Yale::stdlib Yale standard library
This is useful for exploration, but might be too much information. You can also provide a regular expression to search for a particular set of sketches:
cf-sketch> search system
The following sketches match your query:
System::Logrotate Sets defaults and user permissions in the sudoers file
System::Routes Sets defaults and user permissions in the sudoers file
System::Sudoers Sets defaults and user permissions in the sudoers file
System::Syslog Configures syslog
System::access Manage access.conf values
System::config_resolver Configure DNS resolver
System::cron Manage crontab and /etc/cron.d contents
System::etc_hosts Manage /etc/hosts
System::motd Configure the Message of the Day
System::set_hostname Set system hostname. Domain name is also set on Mac, Red Hat and and Gentoo derived distributions (but not Debian).
System::sysctl Manage sysctl values
System::tzconfig Manage system timezone configuration
Use the info
command to get additional information for a sketch, including details
about its parameters:
cf-sketch> info -v System::motd
The following sketches match your query:
Sketch System::motd
Description: Configure the Message of the Day
Authors: Ben Heilman <bheilman@enova.com>
Version: 1.00
License: MIT
Tags: cfdc
Installed: No
Parameters:
For bundle entry
motd: string
motd_path: string
prepend_command: string
dynamic_path: string
symlink_path: string
Step 4. Install a sketch
The first step in using a sketch is to install it. As an example, install
the System::motd
sketch:
cf-sketch> install System::motd
Sketch System::motd installed under /home/vagrant/.cfagent/inputs/sketches.
Sketch README.md installed under System::motd.
Sketch main.cf installed under System::motd.
Sketch params/debian_squeeze.json installed under System::motd.
Sketch params/debian_wheezy.json installed under System::motd.
Sketch params/example.json installed under System::motd.
Sketch params/simple.json installed under System::motd.
Sketch test.cf installed under System::motd.
Verify that the sketch has been installed using the list
command:
cf-sketch> list
The following sketches are installed:
CFEngine::dclib Design Center standard library
CFEngine::stdlib The portions of the CFEngine standard library (also known as COPBL) that are compatible with 3.4.0 releases
System::motd Configure the Message of the Day
Note that the CFEngine::dclib
and CFEngine::stdlib
are
automatically installed as dependencies of the System::motd
sketch.
Step 5. Activate a sketch
Activating sketches includes adding two elements: a parameter set and an environment.
Define the parameter set
Define the parameter set that contains the values that will
be used by the System::motd
sketch:
cf-sketch> define params System::motd
Please enter a name for the new parameter set (default: System::motd-entry-000): motd_params
Querying configuration for parameter set 'motd_params' for bundle 'entry'.
Please enter parameter motd (Message of the Day (aka motd)).
(enter STOP to cancel)
motd : Hello there!
Please enter parameter motd_path (Location of the primary, often only, MotD file).
(enter STOP to cancel)
motd_path [/etc/motd]: /etc/motd
Please enter parameter prepend_command (Command output to prepend to MotD).
(enter STOP to cancel)
prepend_command [/bin/uname -snrvm]: /bin/uname -snrvm
Please enter parameter dynamic_path (Location of the dynamic part of the MotD file).
(enter STOP to cancel)
dynamic_path :
Please enter parameter symlink_path (Location of the symlink to the motd file).
(enter STOP to cancel)
symlink_path :
Defining parameter set 'motd_params' with the entered data.
Parameter set motd_params successfully defined.
Confirm that the parameter set has been properly defined using
the list
command:
cf-sketch> list -v params
The following parameter sets are defined:
motd_params: Sketch System::motd
[System::motd][dynamic_path]:
[System::motd][motd]: Hello there!
[System::motd][motd_path]: /etc/motd
[System::motd][prepend_command]: /bin/uname -snrvm
[System::motd][symlink_path]:
Define an environment
For this example, define an environment that is always active:
cf-sketch> define environment -n walkthrough any
Environment 'walkthrough' successfully defined.
Use the list
command to verify that the environment was
defined:
cf-sketch> list -v env walk
The following environments match your query:
walkthrough
[activated]: any
[test]: !any
[verbose]: !any
Note that the test
and verbose
fields are optional and default to
!any
, which is equivalent to "never" in CFEngine terms.
Run the activate command
Activate the sketch by tying together the sketch name, parameter set, and environment:
cf-sketch> activate System::motd motd_params walkthrough
Using generated activation ID 'System::motd-1'.
Using existing parameter definition 'motd_params'.
Using existing environment 'walkthrough'.
Activating sketch System::motd with parameters motd_params.
Verify that the activation has been created:
cf-sketch> list activations
The following activations are defined:
Activation ID System::motd-1
Sketch: System::motd
Parameter sets: [ motd_params ]
Environment: 'walkthrough'
This means that when the sketches are deployed, the System::motd
sketch will be executed with the values defined in the
motd_params
parameter set, and on the hosts that satisfy the
conditions defined in the walkthrough
environment (which includes
all machines for now).
Step 6. Deploy the sketch: Generate and execute the runfile
So far all the definitions of parameters, environments, and activations
are known only to the cf-sketch
tool. You must deploy these
changes, which creates the appropriate CFEngine policy files for
executing the activated sketches. We have two commands for this:
The run
command allows you to quickly test the execution of the
sketches on the local host. It generates a standalone runfile that
encodes all the necessary information for the activated sketches, and
then executes it using cf-agent
:
cf-sketch> run
Runfile /var/cfengine/inputs/api-runfile-standalone.cf successfully generated.
Now executing the runfile with: /var/cfengine/bin/cf-agent -f /var/cfengine/inputs/api-runfile-standalone.cf
The deploy
command generates a non-standalone runfile that is meant
to be loaded and executed from your main promises.cf
file:
cf-sketch> deploy
Runfile /var/cfengine/inputs/api-runfile.cf successfully generated.
More information
The Design Center framework provides an API that takes care of managing all the backend framework, and cf-sketch offers an "expert" mode in addition to the interactive mode described in these instructions.
Once you are familiar with the basic concepts and want to learn more about how things work internally, visit the advanced discussion on configuring sketches.
Visit the Design Center API for reference.
Once you are ready to start writing Design Center sketches, refer to the sketch structure documentation.
Advanced Walkthrough
This walkthrough illustrates how a Design Center sketch can be found, installed, configured, and executed as policy. Many items are already discussed at Command Line Sketches. This Walkthrough provides a more advanced look at sketches in that it describes internal and backend processes.
Please note the following instructions will work with CFEngine Enterprise as well, but the Mission Portal Design Center App is a better user interface for most of the topics covered here.
Before you Begin
Make certain you have installed all necessary software and have checked out the Design Center repository:
Complete the software requirements
Review the basics concepts of sketches
Check out the Design Center repository
Overview
The following topics are discussed in this Walkthrough:
- Search for sketches with the Design Center API
- Search for sketches with cf-sketch in expert mode
- Search for sketches with cf-sketch in interactive mode
- Install a sketch with the Design Center API
- Install a sketch with cf-sketch in expert mode
- Install a sketch with cf-sketch in interactive mode
- Activate a sketch with the Design Center API
- Activate a sketch with cf-sketch in expert mode
- Activate a sketch with cf-sketch in interactive mode
Generate and execute the runfile
- Generate and execute the runfile with the Design Center API
- Generate and execute the runfile with cf-sketch in expert mode
- Generate and execute the runfile with cf-sketch in interactive mode
Prepare the config.json file
To interact with the Design Center, you must tell its API where things
are installed. Copy the config.json
file to your CFEngine personal
directory:
mkdir ~/.cfagent
cp $CHECKOUT/tools/cf-sketch/config.json ~/.cfagent/dc-api-config.json
Note: A config-root.json
file exists that is intended for privileged
(root) usage. It sets things up under /var/cfengine
but still
expects the checkout under ~/source/design-center
.
Open the config.json file. It contains the following JSON data:
{
log: "STDERR",
log_level: 4,
repolist: [ "~/.cfagent/inputs/sketches" ],
recognized_sources: [ "~/source/design-center/sketches" ],
runfile: { location: "~/.cfagent/inputs/api-runfile.cf", standalone: true, relocate_path: "sketches", filter_inputs: [] },
vardata: "~/.cfagent/vardata.conf",
}
You can change any setting you want but keeping the default values is recommended.
The paths you see are all relative to your home directory. The log
can be set to a file. The log_level
can be lowered to 1 if you want
less noise (the level is set to 4 in this example).
vardata
: is where the Design Center API stores all configurations.
runfile
: describes where the Design Center API will save an
executable policy with all the sketches you have installed and
activated.
repolist
: is where sketches will be installed.
recognized_sources
: is where sketches will be found. It can be
a URL such as
https://github.com/cfengine/design-center/blob/master/sketches/cfsketches.json
.
You can save some typing by entering
export DCJ=~/.cfagent/dc-api-config.json
at the prompt. From that point on, all command-line interaction can
use $DCJ
and it will expand to the filename above.
From this point on, refer to this command
$CHECKOUT/tools/cf-sketch/cf-dc-api.pl
as $CFAPI
for brevity.
You can save some typing by entering
export CFAPI=$CHECKOUT/tools/cf-sketch/cf-dc-api.pl
at the prompt. From that point on, all command-line interaction can
use $CFAPI
and it will expand to the command above.
Search for sketches
This section explains many details that the later sections in the Walkthrough will skip for brevity.
Search for sketches with the Design Center API
This is what all the other Design Center tools use. You do not need to know this protocol or know that it is used.
echo '{ dc_api_version: "3.6.0", request: {search: true } }' | $CFAPI $DCJ
If you get errors here, you might be missing Perl modules or the CFEngine agent. Look at the Design Center Wiki for possible solutions to your problem.
Output:
DCAPI::log3(DCAPI.pm:173): Successfully loaded vardata file /home/tzz/.cfagent/vardata.conf
DCAPI::log(DCAPI.pm:376): Searching location ~/source/design-center/sketches for terms true
DCAPI::Sketch::matches(Repo.pm:118): sketch Applications::Memcached matched terms true
...
DCAPI::Sketch::matches(Repo.pm:118): sketch Webserver::Install matched terms true
All of the above is debugging output. With log
set to a file name,
the output will go to that file. With log_level
set to 1, only the
essential errors will be shown. The actual API response is:
{
"api_ok":
{
"warnings":[],"success":true,"errors":[],"error_tags":{},
"data":
{
"search":
{
"/home/tzz/source/design-center/sketches":
{
"System::config_resolver":"System::config_resolver", ... ,"System::set_hostname":"System::set_hostname"
}
}
},
"log":[],"tags":{}
}
}
The response says (with many sketch names omitted for brevity): Here, these are all my sketches.
Note that this is not what you would use daily.
Search for sketches with cf-sketch
in expert mode
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --expert --cfpath=/var/cfengine/bin --apiconfig $DCJ --search | sort
Output:
Applications::Memcached Sketch for installing, configuring, and starting memcached.
...
You piped the command above through the standard sort
command to
sort the results. You can omit the sort
.
Search for sketches with cf-sketch
in interactive mode
Run:
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --apiconfig $DCJ
You'll see this prompt, or something like it:
Welcome to cf-sketch version 3.6.0.
CFEngine AS, 2013.
Enter any command to cf-sketch, use 'help' for help, or 'quit' or '^D' to quit.
cf-sketch>
Now enter the search
command:
cf-sketch> search
The following sketches are available:
Applications::Memcached Sketch for installing, configuring, and starting memcached.
...
Yale::stdlib Yale standard library
cf-sketch>
This is the easiest method but you might want the expert or direct API interaction for specific purposes. All three are shown throughout this Walkthrough.
Install a sketch
Install a sketch with the Design Center API
echo '{ dc_api_version: "3.6.0", request: {install: {sketch:"System::motd", force:true} } }' | $CFAPI $DCJ
The force
parameter tells the Design Center API to overwrite the
sketch even if it is installed already.
Output:
DCAPI::log3(DCAPI.pm:173): Successfully loaded vardata file /home/tzz/.cfagent/vardata.conf
...
DCAPI::log(DCAPI.pm:576): Installing sketch: {"source":["~/source/design-center/sketches"],"target":"~/.cfagent/inputs/sketches","sketch":"System::motd","force":true}
DCAPI::log4(Repo.pm:179): Installing sketch System::motd: copying /home/tzz/source/design-center/sketches/system/motd/README.md to /home/tzz/.cfagent/inputs/sketches/system/motd/README.md
DCAPI::log4(Repo.pm:179): Installing sketch System::motd: copying /home/tzz/source/design-center/sketches/system/motd/main.cf to /home/tzz/.cfagent/inputs/sketches/system/motd/main.cf
DCAPI::log4(Repo.pm:179): Installing sketch System::motd: copying /home/tzz/source/design-center/sketches/system/motd/params/debian_squeeze.json to /home/tzz/.cfagent/inputs/sketches/system/motd/params/debian_squeeze.json
DCAPI::log4(Repo.pm:179): Installing sketch System::motd: copying /home/tzz/source/design-center/sketches/system/motd/params/debian_wheezy.json to /home/tzz/.cfagent/inputs/sketches/system/motd/params/debian_wheezy.json
DCAPI::log4(Repo.pm:179): Installing sketch System::motd: copying /home/tzz/source/design-center/sketches/system/motd/params/example.json to /home/tzz/.cfagent/inputs/sketches/system/motd/params/example.json
DCAPI::log4(Repo.pm:179): Installing sketch System::motd: copying /home/tzz/source/design-center/sketches/system/motd/params/simple.json to /home/tzz/.cfagent/inputs/sketches/system/motd/params/simple.json
DCAPI::log4(Repo.pm:179): Installing sketch System::motd: copying /home/tzz/source/design-center/sketches/system/motd/test.cf to /home/tzz/.cfagent/inputs/sketches/system/motd/test.cf
After lots of activity (again, remember to drop down to log_level
1 or 0 if you want to skip all these messages) the sketch is installed.
Finally the API returns:
{
"api_ok":
{
"warnings":[],"success":true,"errors":[],"error_tags":{},
"data":
{
"install":
{
"System::motd":
{
"test.cf":"/home/tzz/.cfagent/inputs/sketches/system/motd/test.cf",
"params/debian_squeeze.json":"/home/tzz/.cfagent/inputs/sketches/system/motd/params/debian_squeeze.json",
"README.md":"/home/tzz/.cfagent/inputs/sketches/system/motd/README.md",
"params/example.json":"/home/tzz/.cfagent/inputs/sketches/system/motd/params/example.json",
"params/debian_wheezy.json":"/home/tzz/.cfagent/inputs/sketches/system/motd/params/debian_wheezy.json",
"main.cf":"/home/tzz/.cfagent/inputs/sketches/system/motd/main.cf",
"params/simple.json":"/home/tzz/.cfagent/inputs/sketches/system/motd/params/simple.json"
},
"~/.cfagent/inputs/sketches":{"System::motd":1}
},
"inventory_save":1
},
"log":[],"tags":{"System::motd":1,"installation":8}
}
}
The above output says that System::motd
was installed in
/home/tzz/.cfagent/inputs/sketches/system/motd/
(because my
config.json
says so), and that the sketch inventory was saved
afterwards.
Install a sketch with cf-sketch
in expert mode
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --expert --cfpath=/var/cfengine/bin --install System::motd --apiconfig $DCJ
Output:
Sketch System::motd is already in target repo; you must uninstall it first
So, we need to force it...
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --expert --cfpath=/var/cfengine/bin --install System::motd --apiconfig $DCJ --force
Output: nothing! In expert mode, when everything is OK, nothing is printed. Only the command return code will tell you if everything went well.
So, we need to make it verbose...
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --expert --cfpath=/var/cfengine/bin --install System::motd --apiconfig $DCJ --force --verbose
Output:
... lots of verbose output, including the API interaction, omitted ...
OK: Got successful result: ... the API result is here, omitted for brevity
The cf-sketch
expert mode is a thin layer over the API for testing
and unattended work, so the above verbose output is not really meant
for everyday use.
Install a sketch with cf-sketch
in interactive mode
Run:
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --apiconfig $DCJ
Now enter the uninstall System::motd
and install System::motd
commands, because just installing an already-installed sketch will not do
anything interesting:
cf-sketch> uninstall System::motd
Deactivated System::motd.
Sketch 'System::motd' was uninstalled.
cf-sketch> install System::motd
Sketch System::motd installed under /home/tzz/.cfagent/inputs/sketches.
cf-sketch>
To view verbose output, use
--verbose
with the interactive cf-sketch
call and view the output.
Activate a sketch
Activate a sketch with the Design Center API
We are going to define a run environment, which will tell the Design Center API that we want an activated sketch, not in test mode, and with verbose output:
echo '{ dc_api_version: "3.6.0", request: {define_environment: { walkthrough: { activated: true, test: false, verbose: true } } } }' | $CFAPI $DCJ
Then we will define the parameters for the System::motd
sketch:
echo '{ dc_api_version: "3.6.0", request: {define: { "motd_params": { "System::motd": { "motd": "\\n ! System is under the control of CFEngine, local changes may by overwritten.\\n", "prepend_command": null } } } } }' | $CFAPI $DCJ
and finally, use the run environment and the parameters to activate the sketch:
echo '{ dc_api_version: "3.6.0", request: {activate: { "System::motd": { environment: "walkthrough", params: [ "motd_params" ] } } } }' | $CFAPI $DCJ
Output (omitting log lines and reformatted):
{"api_ok":{"warnings":[],"success":true,"errors":[],"error_tags":{},
"data":{"define_environment":{"walkthrough":1}},
"log":[],"tags":{"walkthrough":1}}}
{"api_ok":{"warnings":[],"success":true,"errors":[],"error_tags":{},
"data":{"define":{"motd_params":1}},
"log":[],"tags":{"motd_params":1}}}
{"api_ok":{"warnings":[],"success":true,"errors":[],"error_tags":{},
"data":{"activate":{"System::motd":{"params":["motd_params"],"environment":"walkthrough"}}},
"log":[],"tags":{"System::motd":1}}}
This tells us that the API has recorded that we want the sketch
System::motd
to run with the run environment walkthrough
and the
parameters motd_params
.
Activate a sketch with cf-sketch
in expert mode
We'll try the simple.json
parameters that come with System::motd
.
You can look at that file; it is the same as the motd_params
above.
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --expert --apiconfig $DCJ --activate System::motd=$CHECKOUT/sketches/system/motd/params/simple.json --verbose --activated
Output:
...
DCAPI::log(DCAPI.pm:1061): Activations for sketch System::motd are now [{"params":["motd_params"],"environment":"walkthrough"},{"params":["parameter definition from /home/tzz/source/design-center/sketches/system/motd/params/simple.json"],"environment":"cf_sketch_testing","target":"/home/tzz/.cfagent/inputs/sketches"}]
This says we now have two activations. One from the API call above and one we just created. Let's undo the activations:
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --expert --apiconfig $DCJ --deactivate-all
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --expert --apiconfig $DCJ --activate System::motd=$CHECKOUT/sketches/system/motd/params/simple.json --verbose --activated
Output:
...
DCAPI::log(DCAPI.pm:1403): Deactivating all activations: {"System::motd":[{"params":["motd_params"],"environment":"walkthrough"},{"params":["parameter definition from /home/tzz/source/design-center/sketches/system/motd/params/simple.json"],"environment":"cf_sketch_testing","target":"/home/tzz/.cfagent/inputs/sketches"}]}
...
OK: Got successful result: {"success":true,"warnings":[],"errors":[],"error_tags":{},"log":[],"data":{"activate":{"System::motd":{"environment":"cf_sketch_testing","params":["parameter definition from /home/tzz/source/design-center/sketches/system/motd/params/simple.json"],"target":"/home/tzz/.cfagent/inputs/sketches"}}},"tags":{"System::motd":1}}
Looks like it worked! The cf_sketch_testing
environment is created
by cf-sketch
on the fly and will include the same things as the
walkthrough
run environment. The --activated
and --verbose
flags turn on the environment activated
and verbose
flags.
There's also a --test
flag, but we will not use it here.
Activate a sketch with cf-sketch
in interactive mode
Run:
$CHECKOUT/tools/cf-sketch/cf-sketch.pl
Then:
cf-sketch> define params System::motd
Please enter a name for the new parameter set (default: System::motd-entry-000): motd_params
Querying configuration for parameter set 'motd_params' for bundle 'entry'.
Please enter parameter motd (Message of the Day (aka motd), ).
motd : Hello there!
Please enter parameter motd_path (Location of the primary, often only, MotD file, ).
motd_path [/etc/motd]: /etc/motd
Please enter parameter prepend_command (Command output to prepend to MotD, ).
prepend_command [/bin/uname -snrvm]: /bin/uname -snrvm
Please enter parameter dynamic_path (Location of the dynamic part of the MotD file, ).
dynamic_path : null
Please enter parameter symlink_path (Location of the symlink to the motd file, ).
symlink_path : null
Defining parameter set 'motd_params' with the entered data.
Parameter set motd_params successfully defined.
cf-sketch> activate System::motd motd_params walkthrough
Using existing parameter definition 'motd_params'.
Using existing environment 'walkthrough'.
Activating sketch System::motd with parameters motd_params.
...
DCAPI::log(DCAPI.pm:1061): Activations for sketch System::motd are now [{"params":["parameter definition from /home/tzz/source/design-center/sketches/system/motd/params/simple.json"],"environment":"cf_sketch_testing","target":"/home/tzz/.cfagent/inputs/sketches"},{"params":["motd_params"],"environment":"walkthrough","target":"/home/tzz/.cfagent/inputs/sketches","identifier":"System::motd-1"}]
cf-sketch>
As you can see, the expert and interactive modes have completely different usage patterns.
Generate and execute the runfile
Generate and execute the runfile with the Design Center API
Generating the runfile is easy:
echo '{ dc_api_version: "3.6.0", request: {regenerate: { } } }' | $CFAPI $DCJ
Output:
DCAPI::log(DCAPI.pm:249): Saving runfile /home/tzz/.cfagent/inputs/api-runfile.cf
{"api_ok":{"warnings":[],"success":true,"errors":[],"error_tags":{},"data":{},"log":[],"tags":{}}}
Note there is no user control over the location of the runfile, it's
entirely defined in the API's config.json
file. This is by design.
Time to run the policy!!! We know the name of the runfile, so we can run it.
cf-agent -KI -f ~/.cfagent/inputs/api-runfile.cf
We are not in test mode, so we will get errors if we run as a non-privileged user.
2013-06-04T20:12:04-0400 info: This agent is not bootstrapped
2013-06-04T20:12:04-0400 info: Running full policy integrity checks
2013-06-04T20:12:04-0400 error: Unable to open destination file '/etc/motd.cf-after-edit' for writing. (fopen: Permission denied)
2013-06-04T20:12:04-0400 error: /cfsketch_run/methods/'___001_System_motd_entry'/cfdc_motd:entry/files/'$(main_path)': Unable to save file '/etc/motd' after editing
2013-06-04T20:12:04-0400 error: chmod failed on '/etc/motd'. (chmod: Operation not permitted)
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: System::motd license = MIT
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: System::motd dependencies = CFEngine::dclib, CFEngine::stdlib
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: System::motd version 1.00 by Ben Heilman <bheilman@enova.com> starting up...
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: imported environment 'cf_sketch_testing' var 'activated' with value '1'
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: imported environment 'cf_sketch_testing' var 'test' with value ''
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: imported environment 'cf_sketch_testing' var 'verbose' with value '1'
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: imported environment 'cf_sketch_testing' class 'activated' because 'default:runenv_cf_sketch_testing_activated' was defined
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: imported environment 'cf_sketch_testing' class 'verbose' because 'default:runenv_cf_sketch_testing_verbose' was defined
2013-06-04T20:12:04-0400 notice: R: cfdc_motd:entry: running in verbose mode
Complete.
Generate and execute the runfile with cf-sketch
in expert mode
$CHECKOUT/tools/cf-sketch/cf-sketch.pl --expert --apiconfig $DCJ --generate
Output:
...
DCAPI::log(DCAPI.pm:249): Saving runfile /home/tzz/.cfagent/inputs/api-runfile.cf
Run the runfile:
cf-agent -KI -f ~/.cfagent/inputs/api-runfile.cf
The output will be the same as in the previous section.
Generate and execute the runfile with cf-sketch
in interactive mode
Run:
$CHECKOUT/tools/cf-sketch/cf-sketch.pl
You'll see:
cf-sketch> generate
...
DCAPI::log(DCAPI.pm:249): Saving runfile /home/tzz/.cfagent/inputs/api-runfile.cf
Runfile /home/tzz/.cfagent/inputs/api-runfile.cf successfully generated.
Run it:
cf-agent -KI -f ~/.cfagent/inputs/api-runfile.cf
The output will be the same as in the previous section.
Write a new Sketch
Enterprise and Community Users can Write Sketches
Overview
This page describes how to create a Design Center sketch from a basic CFEngine agent bundle. In effect the sketch is a wrapper, taking parameters from the Design Center API or the CFEngine Enterprise Mission Portal, and passing them down to the agent bundle.
The Bundle
The bundle we will wrap with a sketch is a simple use of the users
promise type:
bundle agent ensure_users(users, group, homedir, shell)
{
users:
"$(users)" -> {"PCI-DSS-2", "Baseline_developers_2_3"}
policy => "present",
home_dir => "$(homedir)/$(users)",
group_primary => $(group),
shell => $(shell),
handle => "ensure_user_setup",
home_bundle => setup_home_dir($(users), $(homedir));
}
bundle agent setup_home_dir(user, group, homedir)
{
files:
"$(homedir)/$(user)/." create => "true";
}
If you don't understand what it does, please look at Tutorials and the CFEngine Guide to learn more about CFEngine's policy language, syntax, and operation. We will not worry about the internals except to apply namespaces and testing guards.
The sketch.json Metadata
The first step is to define the sketch metadata. This is the annoying
administrivia that makes a package system such as Design Center
useful. Simply take an existing sketch.json
file such as the one below and modify it:
{
manifest:
{
"main.cf": { description: "main file" },
"README.md": { documentation: true },
},
metadata:
{
name: "System::Users",
description: "Configure users with parameters",
version: "1.00",
license: "MIT",
tags: [ "cfdc", "users", "enterprise_compatible", "enterprise_3_6" ],
authors: [ "Ted Zlatanov <tzz@lifelogs.com>" ],
depends: { "CFEngine::sketch_template": {}, cfengine: { version: "3.6.0" }, os: [{ "linux": "Linux", "solaris": "Solaris", "aix": "AIX", "windows": "Windows" }] }
},
api:
{
// this is the name of the bundle!
ensure_users:
[
{ type: "bundle_options", name: "Ensure the users exist as specified" },
{ type: "environment", name: "runenv", },
{ type: "metadata", name: "mymetadata", },
{ type: "list", name: "users", validation: "LIST_OF_STRING_NONEMPTY", description: "User names to add (separate by commas)" },
{ type: "string", name: "group", validation: "STRING_NONEMPTY", description: "Primary user group" },
{ type: "string", name: "homedir", default: "/home", validation: "PATH_ABSOLUTE_UNIX_OR_WINDOWS", description: "Location of the user's home directory" },
{ type: "string", name: "shell", default: "/bin/bash", choice: [ "/bin/sh", "/bin/bash", "/bin/csh", "/bin/tcsh", "/bin/zsh" ], description: "User shell" },
],
},
namespace: "cfdc_users",
interface: [ "main.cf" ],
}
- add any files you distribute with the sketch to the
manifest
- set the
authors
,tags
,name
,description
, etc. metadata - define an API, which mirrors the bundle we have. Here we do a few things extra:
- add
bundle_options
with the name of the bundle we want to show - add
environment
andmetadata
parameters, which carry the "glue" between Design Center and CFEngine - bring in the
users
,group
,homedir
, andshell
parameters with atype
, adefault
if needed, avalidation
or achoice
as needed, and adescription
- add
- set the
namespace
as shown to ensure this bundle won't conflict with others - set the
interface
to the list of files that have to be included by CFEngine for the sketch to work, normally justmain.cf
main.cf: The Converted Bundle
Now main.cf
will start with the original bundle, but we'll modify it.
body file control
{
namespace => "cfdc_users";
}
bundle agent ensure_users(runenv, metadata, users, group, homedir, shell)
{
#@include "REPO/sketch_template/standard.inc"
users:
!dc_test::
"$(users)" -> {"PCI-DSS-2", "Baseline_developers_2_3"}
policy => "present",
home_dir => "$(homedir)/$(users)",
group_primary => $(group),
shell => $(shell),
handle => "ensure_user_setup",
home_bundle => setup_home_dir($(users), $(homedir));
reports:
dc_verbose.dc_test::
"$(dcbundle): simulating user = $(users) with group $(group), home dir $(homedir) and shell $(shell)";
dc_verbose.!dc_test::
"$(dcbundle): ensuring user = $(users) with group $(group), home dir $(homedir) and shell $(shell)";
}
bundle agent setup_home_dir(user, group, homedir)
{
files:
"$(homedir)/$(user)/." create => "true";
}
- add a
namespace
matchingsketch.json
- add the
#@include
statement which sets up the Design Center machinery - add the
runenv
andmetadata
parameters (note you don't use them!) - report on what you'll do, using the classes
dc_verbose
anddc_test
which mean "you're in verbose mode" and "you're in test mode" respectively - only make the users if not in
dc_test
mode
Package The Sketch
This part is really too easy. We should make it harder so you have something to complain about!
There are two steps:
- put your files in a directory (say
/my/repo/sketches/xyz
if your sketches will live under/my/repo
and the one you made is calledxyz
). This is just the files from the manifest:README.md
, which you could cut from the manifest or auto-generate, is just a README filemain.cf
which you just saw above- plus
sketch.json
as shown above
- regenerate the sketch index for
/my/repo
and install your sketch into/var/cfengine/design-center/sketches
(the "live" repository of sketches). Run the following commands:
cp -rp /var/cfengine/share/*Base/sketches/sketch_template /my/repo/sketches/
/var/cfengine/design-center/bin/cf-sketch --make_cfsketches --inputs /my/repo --is=/my/repo/sketches/cfsketches.json
/var/cfengine/design-center/bin/cf-sketch --make_readme --is=/my/repo/sketches/cfsketches.json
/var/cfengine/design-center/bin/cf-sketch --install-all --is=/my/repo/sketches/cfsketches.json --inputs=/var/cfengine/design-center
See Maintaining your own sketch repository for these exact commands to run, with a longer explanation for each one.
You're Done!
That's all there is to writing a sketch. You should now look at
sketchify - Write A New Sketch From An Existing Bundle
for a guide to using the sketchify
tool. See
Maintaining your own sketch repository
to find out how to create your own sketch repository and install sketches from it.
Your users can then use the sketch you've written as described in Deploy your first Policy.
sketchify - Write A New Sketch From An Existing Bundle
Enterprise and Community Users can Write Sketches from the Command Line
Overview
This page describes how to create a Design Center sketch by converting an existing policy
into a sketch and by using the sketchify
tool in cf-sketch
to complete the process.
These steps are followed:
Step 1. Select a policy to convert into a sketch
Step 3. Define the sketch interface
Step 4. Revise the policy file as necessary
Step 5. Use the sketchify command to wrap the policy file into a sketch structure
Step 6. Verify that the new sketch is ready for installation and use
Refer to the password_expiration()
bundle example as you
follow the steps outlined on this page.
This information is from Diego Zamboni's book, Learning CFEngine 3. Used with permission. It has been updated, reformatted and edited for website consistency.
Before you Begin
This is an advanced topic; we assume that you know how to write CFEngine policy, and that you are familiar with the Design Center command line tools and, if you are an Enterprise user, with the Design Center UI.
Instructions
Step 1. Select a policy to convert into a sketch
The foundation of any Design Center sketch should be a working piece of CFEngine
policy in the form of a bundle of type agent that performs the appropriate functionality.
This bundle can call other bundles or bodies as appropriate, but it should be callable
as a single point of entry. Until you become more familiar with how sketches
are structured, write your bundles first as regular CFEngine
policy, and then convert them to sketches. The instructions on this page create a sketch from the
password_expiration()
bundle.
Step 2. Define a sketch name
Arbitrary names are acceptable, but
the Design Center by convention encourages us to use names of the form Category::Sketch
,
or even Category::Subcategory::Sketch
. For our password-expiration configuration
sketch, use Security::password_expiration
.
Step 3. Define the sketch interface
Identify and name configurable parameters
In our original example, all the
parameters are specified as variables inside the password_expiration()
bundle. For
a sketch, however, we want those values as parameters to be specified by the user when they
configure the sketch. The next step, then, is to look through the original code, make a
list of what those configurable parameters should be, and decide on their names:
pass_max_days
The maximum password age in days.
pass_min_days
The minimum password age, also in days.
pass_warn_age
The warning period before a password expires, in days.
min_uid
The minimum UID for setting password-expiration parameters. Users with UID below this threshold will not be modified.
skipped_users
A comma-separated list of usernames to skip when setting password-expiration parameters.
skipped_uids
A comma-separated list of UIDs to skip when setting password-expiration parameters.
All of these parameters can be specified as strings, just as they are in the original policy code.
Create a namespace
You must also decide on a namespace in which to place the sketch. Namespaces are top-level naming divisions that help avoid conflicts in bundle, body, or class names. Use a namespace that contains the following:
- a reference to the origin of the sketch. For example, all CFEngine-produced sketches have namespaces that start with cfdc_ for CFEngine Design Center
- the name of the sketch, or a shortened, representative version of it.
For this example, use cflearn_password_expiration.
Step 4. Revise the policy file as necessary
Once the sketch interface is defined, rewrite the policy file as necessary to make it ready to use as a sketch. Below is the updated code, with some comments about the changes made As you go through these, compare them to the original code in the password_expiration() bundle:
bundle agent password_expiration(pass_max_days, pass_min_days, pass_warn_age,
min_uid, skipped_users, skipped_uids) # <1>
{
vars:
# We store the individual parameters in an array,
# for easier reference and file editing
"logindefs[PASS_MAX_DAYS]" string => "$(pass_max_days)"; # <2>
"logindefs[PASS_MIN_DAYS]" string => "$(pass_min_days)";
"logindefs[PASS_WARN_AGE]" string => "$(pass_warn_age)";
# Position of each parameter in /etc/shadow
"fieldnum[PASS_MIN_DAYS]" string => "4";
"fieldnum[PASS_MAX_DAYS]" string => "5";
"fieldnum[PASS_WARN_AGE]" string => "6";
# List of parameters to modify
"params" slist => getindices("logindefs");
# Get list of users, and also generate them in canonified form
# This list already excludes users specified by UID or name.
"users" slist => getusers("$(skipped_users)", "$(skipped_uids)");
"cusers[$(users)]" string => canonify("$(users)");
classes:
# Define classes for users that must not be modified by UID threshold
"skip_$(cusers[$(users)])" expression => islessthan(getuid("$(users)"),
"$(min_uid)");
files:
linux:: # <3>
"/etc/login.defs"
handle => "edit_logindefs",
comment => "Set desired login.defs parameters",
edit_line =>
default:set_config_values(
"cflearn_password_expiration:password_expiration.logindefs"); # <4>
"/etc/shadow"
handle => "edit_shadow_$(params)",
comment => "Modify $(params) for individual users.",
edit_defaults => default:backup_timestamp, # <5>
edit_line => default:set_user_field("$(users)",
"$(fieldnum[$(params)])",
"$(logindefs[$(params)])"),
ifvarclass => "!skip_$(cusers[$(users)])";
reports:
!linux:: # <6>
"Warning: Security::password_expiration only works on Linux for now.";
}
Point by Point:
The logic of the code has not changed, but a few things have been updated or rearranged:
<1> We have added all the configurable parameters we determined earlier as arguments
to our password_expiration()
bundle. All of these values are now accepted as arguments
instead of being hardcoded into the policy. This is the entry point for our sketch.
<2> We use the new parameters throughout the code, instead of the hard-coded values we had before.
<3> We have added a class expression to limit the execution of the sketch to systems
that support its behavior. This is necessary because a sketch might be activated on
many different systems, and it needs to do the right thing regardless of
where it is running. In this case, we have limited it to Linux systems, in which we
know the password-expiration
parameters are configured using the /etc/login.defs
file.
<4> Here is the first use of namespaces, in two places: We have added the
default:
namespace specification to the standard library bundle
set_config_values()
, and we have specified our sketch namespace in the fully-qualified
name of the logindefs
array that we pass to set_config_values()
. The
fully-qualified name of the array (cflearn_password_expiration:password_expiration.logindefs
)
contains the namespace, the bundle name, and the array name.
<5> We must add the default:
namespace to all the standard library components we
use and, in this case, also to the backup_timestamp
body and the set_user_field()
bundle.
<6> Finally, and to complement the limitation of functionality of the sketch to Linux
systems, we have added a reports:
promise that prints a warning on non-Linux systems to let
users know that the sketch is non-functional on them.
We now have the policy file in a shape that is well suited for conversion into a sketch.
Step 5. Use the sketchify command to wrap the policy file into a sketch structure
The next step is to actually wrap the revised policy file into the appropriate structure required
by a sketch, which includes putting the file into its own directory. Add to that directory
a README
file and a file named sketch.json
that contains all the metadata about the
sketch, as well as all the information needed to configure and invoke it. You can find
the full specification in the
Writing a Design Center Sketch
guide, but you can also
use the sketchify
command in cf-sketch
to do it automatically. The sketchify
command reads the
policy file, asks you for the appropriate information, and produces a ready-to-use sketch
in your local checkout of the Design Center repository.
The sketchify
command takes as its only argument the file that contains our policy file,
which it reads and analyzes for bundles of type agent
. Our policy file contains only one
bundle, so it is used automatically as the entry point for the sketch (if more than one
agent bundle is found, you will be asked which one you want to use as the sketch entry
point):
# /var/cfengine/design-center/bin/cf-sketch sketchify /vagrant/password_expiration.cf
Reading file '/vagrant/password_expiration.cf'.
Automatically choosing the only agent bundle in /vagrant/password_expiration.cf:
'password_expiration'
I will now prompt you for the data needed to generate the sketch.
Please enter STOP at any prompt to interrupt the process.
Note: The Design Center framework supports sketches with more than one entry point, but sketchify as of this writing lets you choose only one of them.
Next, sketchify asks for general information about the sketch, including its name, description, version number, license (most sketches in the Design Center use the MIT license), tags, and author information. You can also enter the names of other CFEngine policy files that should be included in this sketch. Most sketches are contained in a single .cf file, but if you have a very complex sketch, the ability to package multiple .cf files within the same sketch could be useful:
Sketch name: Security::password_expiration
One-line description for the new sketch: Manage password expiration and warning periods
Sketch version number: 1.0
Sketch license: MIT
Sketch tags (comma-separated list): security,cflearn,passwords
Authors (comma-separated list, preferably of the form 'Name <email>'): Diego Zamboni <diego.zamboni@cfengine.com>
Please enter any other files that need to be included with this sketch (press Enter to stop):
Note: This has nothing to do with sketch dependencies—any file(s) you specify
here will be included within the sketch you are creating. As of this writing,
sketchify does not handle sketch dependencies. You must include
them by hand in the generated sketch.json
file.
Next, sketchify
queries us for the information needed for defining
the sketch API. For each parameter of the entry bundle, sketchify
prompts for its type, a description, and a optional default and
example values (the example value is shown in the Design Center app in
CFEngine Enterprise). In our example, we give default values for all
the parameters except skipped_users
and skipped_uids
:
Thank you. I will now prompt you for the information regarding the parameters
of the entry point for the sketch.
For each parameter, you need to provide a type, a description, and optional default and example values.
(enter STOP at any prompt to abort)
For parameter 'pass_max_days':
Type [(1) string, (2) boolean, (3) list, (4) array]: : 1
Short description: : Maximum password age in days
Default value (empty for no default): : 180
Example value (empty for no example): : 180
For parameter 'pass_min_days':
Type [(1) string, (2) boolean, (3) list, (4) array]: : 1
Short description: : Minimum password age in days
Default value (empty for no default): : 5
Example value (empty for no example): : 5
For parameter 'pass_warn_age':
Type [(1) string, (2) boolean, (3) list, (4) array]: : 1
Short description: : Warning period before password expires, in days
Default value (empty for no default): : 2
Example value (empty for no example): : 2
For parameter 'min_uid':
Type [(1) string, (2) boolean, (3) list, (4) array]: : 1
Short description: : Minimum UID to consider when updating existing accounts
Default value (empty for no default): : 500
Example value (empty for no example): : 500
For parameter 'skipped_users':
Type [(1) string, (2) boolean, (3) list, (4) array]: : 1
Short description: : Comma-separated list of usernames to skip when updating existing accounts
Default value (empty for no default): :
Example value (empty for no example): : diego,joe
For parameter 'skipped_uids':
Type [(1) string, (2) boolean, (3) list, (4) array]: : 1
Short description: : Comma-separated list of UIDs to skip when updating existing accounts
Default value (empty for no default): :
Example value (empty for no example): : 550,1027
We are done with the API!
Having defined the sketch API, sketchify now queries you for information about the namespace to use for this sketch. We decided before which namespace to use, but the namespace declaration does not yet appear in the policy file we are using, so sketchify offers to insert it automatically:
Now checking the namespace declaration.
The file '/vagrant/password_expiration.cf' does not have a namespace declaration.
It is recommended that every sketch has its own namespace to avoid potential naming conflicts with other sketches or policies.
I can insert the appropriate namespace declaration, and have generated a suggested namespace for you: cfdc_security_password_expiration
Please enter the namespace to use for this sketch: : cfdc_security_password_expiration
Note: If you insert the namespace declaration in the policy file by hand, before running it through sketchify, the command will automatically detect and use the declaration.
In addition to the parameters defined in the API, a sketch entry bundle can receive two
special parameters of type environment
and metadata
. If used, these parameters will be
automatically generated and passed by the Design Center framework when executing
the sketch.
The
environment
parameter contains the name of the environment with which the sketch has been activated. This allows the sketch to access the characteristics of the environment, including the verbose and testing fields (interpreted as classes) so that the sketch can easily use them as conditions to alter its behavior.The
metadata
parameter contains the name of an array in which the Design Center framework automatically stores all the sketch metadata, including its name and description, authors, etc.
If these parameters are not already passed to the entry bundle in the input file, sketchify will ask you if you want to add them:
The entry point 'password_expiration' doesn't seem to receive
parameters of type 'environment' or 'metadata'.
These arguments are useful for the sketch to respond to different run
environment parameters (i.e. test or verbose mode) or to have access
to its own metadata. I can automatically add these parameters to the
bundle, together with some boilerplate code to put their information
in classes and variables.
Would you like me to add environment/metadata parameters and code to
the sketch? (Y/n) : y
In addition to adding the parameters to the bundle, sketchify also adds some boilerplate code to do the following:
Extract the values of all fields defined in the active environment (at least
activated
,verbose
, andtesting
, and possibly others if defined) into both classes and variables. For example, it will create a string variable namedverbose
that contains the class expression stored in that field, and also a class namedverbose
that will be set to the result of evaluating that class expression. You can then use that class within your sketch to easily enable additional reports when verbose mode has been activated in the current environment.Add some other information for better integration of the sketch into the CFEngine Enterprise Design Center app.
As of this writing, the following code is automatically inserted by sketchify at the top
of the bundle. This line is automatically expanded into the contents
of the template file which can be found at
/var/cfengine/design-center/sketches/sketch_template/standard.inc
.
#@include "REPO/sketch_template/standard.inc"
sketchify
now asks you for the location under the currently-used
sketch repository where the new sketch should be stored:
Thank you! We are almost done.
Please enter the directory where the new sketch will be stored.
If you enter a relative path, it will be used within the currently configure sketch repository (/var/cfengine/design-center/sketches). If you enter an absolute path, it will be used as-is. The directory will be created if needed.
I have generated a suggestion based on your sketch name: security/password_expiration
Directory: security/password_expiration
Before writing the sketch, sketchify
shows you a menu with all the
parameters you entered, and gives you a chance to modify them. If you
made any mistakes or want to change anything, enter the number of the
corresponding parameter and sketchify
will prompt you for the values
again.
You now have a chance to modify any of the information you entered.
These are the current sketch parameters:
1. Sketch name: Security::password_expiration
2. One-line description for the new sketch: Manage password expiration and warning periods
3. Sketch version number: 1.0
4. Sketch license: MIT
5. Sketch tags: cflearn, enterprise_compatible, passwords, security, sixified, sketchify_generated
6. Authors: Diego Zamboni <diego.zamboni@cfengine.com>
7. Extra manifest files:
8. Sketch API:
For bundle password_expiration
pass_max_days: string (Maximum password age in days) [default value: '180']
pass_min_days: string (Minimum password age in days) [default value: '5']
pass_warn_age: string (Warning period before password expires, in days) [default value: '2']
min_uid: string (Minimum UID to consider when updating existing accounts) [default value: '500']
skipped_users: string (Comma-separated list of users to skip when updating existing accounts)
skipped_uids: string (Comma-separated list of UIDs to skip when updating existing accounts)
9. Namespace: cfdc_security_password_expiration
10. Runenv and metadata parameters: Environment and metadata parameters and boilerplate code WILL be added
11. Output directory: /var/cfengine/design-center/sketches/
Please enter the number of the part you want to modify (1-11, Enter to
continue)
Finally, when you press "Enter" in the prompt above, sketchify
writes all the files for the sketch in the appropriate directory:
Your new sketch will be stored under /var/cfengine/design-center/sketches/security/password_expiration
Writing /var/cfengine/design-center/sketches/security/password_expiration/sketch.json
Transferring /vagrant/password_expiration.cf to /var/cfengine/design-center/sketches/security/password_expiration/password_expiration.cf
Regenerating sketch index in /var/cfengine/design-center/sketches
Generating a README file for the new sketch.
We are done! Please check your new sketch under
/var/cfengine/design-center/sketches/security/password_expiration.
The sketch is created; the process is complete.
Step 6. Verify that the new sketch is ready for installation and use
Verify this using cf-sketch
. Search for the password sketch:
cf-sketch> search password
The following sketches match your query:
Security::password_expiration Manage password expiration and warning periods
cf-sketch> install Security::password_expiration
Sketch Security::password_expiration installed under
/var/cfengine/masterfiles/sketches.
cf-sketch> info -v Security::password_expiration
The following sketches match your query:
Sketch Security::password_expiration
Description: Manage password expiration and warning periods
Authors: Diego Zamboni <diego.zamboni@cfengine.com>
Version: 1.0
License: MIT
Tags: passwords, security, sketchify_generated, cflearn
Installed: Yes, under /var/cfengine/masterfiles/sketches
Activated: No
Parameters:
For bundle password_expiration
pass_max_days: string (Maximum password age in days) [default value: '180']
pass_min_days: string (Minimum password age in days) [default value: '5']
pass_warn_age: string (Warning period before password expires, in days)
[default value: '2']
min_uid: string (Minimum UID to consider when updating existing accounts)
[default value: '500']
skipped_users: string (Comma-separated list of usernames to skip when updating existing accounts)
skipped_uids: string (Comma-separated list of UIDs to skip when updating existing accounts)
While sketchify automates most of the process of creating a sketch from an existing bundle, it cannot handle a few items. Thus, look at the files it generates for sanity checking. Here are some of the things you might want or need to fix by hand:
Dependencies: If your sketch depends on other sketches, you must add them by hand to the
depends
metadata element in the generated sketch.json file. At the moment, sketchify automatically inserts a dependency on CFEngine 3.5.0, which is the minimum recommended version of using Design Center sketches.Multiple entry points: The Design Center framework supports multiple entry points per sketch (to different bundles). This is not supported at the moment by sketchify, so you must add any additional entry points by hand.
Calls to standard library bundles and bodies need to be prefixed with
default:
so that they are correctly found when called from the sketch namespace.
Example
password_expiration() bundle
Below is the existing policy example that is used to turn into a sketch. Refer to it as you follow the steps for creating a sketch.
bundle agent password_expiration
{
vars:
# Maximum password age
"logindefs[PASS_MAX_DAYS]" string => "180";
# Minimum password age (minimum days between changes)
"logindefs[PASS_MIN_DAYS]" string =>"10";
# Warning period (in days) before password expires
"logindefs[PASS_WARN_AGE]" string => "5";
# Position of each parameter in /etc/shadow
"fieldnum[PASS_MIN_DAYS]" string => "4";
"fieldnum[PASS_MAX_DAYS]" string => "5";
"fieldnum[PASS_WARN_AGE]" string => "6";
# List of parameters to modify
"params" slist => getindices("logindefs");
# UIDs below this threshold will not be touched
"uidthreshold" int => "500";
# Additionally, these users and UIDs will not be touched.
# These are comma-separated lists.
"skipped_users" string => "vboxadd,nobody";
"skipped_uids" string => "1000,1005";
# Get list of users, and also generate them in canonified form
"users" slist => getusers("$(skipped_users)", "$(skipped_uids)");
"cusers[$(users)]" string => canonify("$(users)");
classes:
# Define classes for users that must not be modified,
# either by UID threshold or by username
"skip_$(cusers[$(users)])" expression => islessthan(getuid("$(users)"),
"$(uidthreshold)");
files:
"/etc/login.defs"
handle => "edit_logindefs",
comment => "Set desired login.defs parameters",
edit_line => set_config_values("password_expiration.logindefs");
"/etc/shadow"
handle => "edit_shadow_$(params)",
comment => "Modify $(params) for individual users.",
edit_defaults => backup_timestamp,
edit_line => set_user_field("$(users)",
"$(fieldnum[$(params)])",
"$(logindefs[$(params)])"),
ifvarclass => "!skip_$(cusers[$(users)])";
}
Reporting
No promises made in CFEngine imply automatic aggregation of data to a central location. In CFEngine Enterprise (our commercial version), an optimized aggregation of standardized reports is provided, but the ultimate decision to aggregate must be yours.
Monitoring and reporting capabilities in CFEngine depend on your installation:
Enterprise Edition Reporting
The CFEngine Enterprise edition offers a framework for configuration management that goes beyond building and deploying systems. Features include compliance management, reporting and business integration, and tools for handling the necessary complexity.
In a CFEngine Enterprise installation, the CFEngine Server aggregates information about the environment in a centralized database. By default data is collected every 5 minutes from all bootstrapped hosts and includes information about:
- logs about promises kept, not kept and repaired
- current host contexts and classifications
- variables
- software information
- file changes
This data can be mined using SQL queries and then used for inventory management, compliance reporting, system diagnostics, and capacity planning.
Access to the data is provided through:
Command-Line Reporting
Community Edition
Basic output to file or logs can be customized on a per-promise basis. Users can design their own log and report formats, but data processing and extraction from CFEngine's embedded databases must be scripted by the user.
Note:
If you have regular reporting needs, we recommend using our commercially-supported version of CFEngine, Enterprise. It will save considerable time and resources in programming, and you will have access to the latest developments through the software subscription.
Monitoring and Reporting
What are Monitoring and Reporting?
Monitoring is the sampling of system variables at regular intervals in order to present an overview of actual changes taking place over time. Monitoring data are often presented as extensive views of moving-line time series. Monitoring has the ability to detect anomalous behavior by comparing past and present.
The term reporting is usually taken to mean the creation of short summaries of specific system properties suitable for management. System reports describe both promises about the system, such as compliance, discovered changes and faults.
The challenge of both these activities is to compare intended or promised, behavior with the actual observed behavior of the system.
Should Monitoring and Configuration be Separate?
The traditional view of IT operations is that configuration, monitoring, and reporting are three different things that should not be joined. Traditionally, all three have been independent centralized processes. This view has emerged historically, but it has a major problem: Humans are needed to glue these parts back together. Monitoring as an independent activity is inherently non-scalable. When numbers of hosts grow beyond a few thousands, centralized monitoring schemes fail to manage the information. Tying configuration (and therefore repair) to monitoring at the host level is essential for the effective management of large and distributed data facilities. CFEngine foresaw this need in 1998, with its Computer Immunology initiative, and continues to develop this strategy.
CFEngine's approach is to focus on scalability. The commercial editions of
CFEngine provide what meaningful information they can in a manner that can
be scaled to tens of thousands of machines.
<!--- End include: /home/jenkins/workspace/build-documentation-3.6/label/DOCUMENTATION_x86_64_linux_debian_6/documentation/guide/reporting/monitoring-reporting.markdown
-->
Command-Line Reports
Command-line reporting is available to Enterprise and Community users.
Overview
The following report topics are included:
CFEngine output levels
CFEngine's default behavior is to report to the console (known as standard output). It's default behavior is to report nothing except errors that are judged to be of a critical nature.
By using CFEngine with the inform flag, you can alter the default to report on action items (actual changes) and warnings:
# cf-agent -I
# cf-agent --inform
By using CFEngine with the verbose flag, you can alter the default to report all of its thought-processes. You should not interpret a message that only appears in CFEngine's verbose mode as an actual error, only as information that might be relevant to decisions being made by the agent:
# cf-agent -v
# cf-agent --verbose
Creating custom reports
CFEngine allows you to use reports
promises to make reports of your own. A simple
example of this is shown below.
body common control
{
bundlesequence => { "test" };
}
#
bundle agent test
{
reports:
cfengine_3::
"$(sys.date),This is a report"
report_to_file => "/tmp/test_log";
}
We can apply this idea to make more useful custom reports. In this example, the agent tests for certain software package and creates a simple HTML file of existing software:
body common control
{
bundlesequence => { "test" };
}
#
bundle agent test
{
vars:
"software" slist => { "gpg", "zip", "rsync" };
classes:
"no_report" expression => fileexists("/tmp/report.html");
"have_$(software)" expression => fileexists("/usr/bin/$(software)");
reports:
no_report::
"
<html>
Name of this host is: $(sys.host)<br>
Type of this host is: $(sys.os)<br>
"
report_to_file => "/tmp/report.html";
#
"
Host has software $(software)<br>
"
ifvarclass => "have_$(software)",
report_to_file => "/tmp/report.html";
#
"
</html>
"
report_to_file => "/tmp/report.html";
}
The outcome of this promise is a file called /tmp/report.html which contains the following output:
<html>
Name of this host is: atlas<br>
Type of this host is: linux<br>
Host has software gpg<br>
Host has software zip<br>
Host has software rsync<br>
</html>
The mechanism shown above can clearly be used to create a wide variety of report formats, but it requires a lot of coding and maintenance by the user.
Including data in reports
CFEngine generates information internally that you might want to use in reports.
For example, the agent cf-agent
interfaces with the local light-weight monitoring agent
cf-monitord
so that system state can be reported simply:
body common control
{
bundlesequence => { "report" };
}
###########################################################
bundle agent report
{
reports:
linux::
"/etc/passwd except $(const.n)"
showstate => { "otherprocs", "rootprocs" };
}
A bonus to this is that you can get CFEngine to report system anomalies:
reports:
rootprocs_high_dev2::
"RootProc anomaly high 2 dev on $(mon.host) at approx $(mon.env_time)
measured value $(mon.value_rootprocs)
average $(mon.average_rootprocs) pm $(mon.stddev_rootprocs)"
showstate => { "rootprocs" };
entropy_www_in_high&anomaly_hosts.www_in_high_anomaly::
"High entropy incoming www anomaly on $(mon.host) at $(mon.env_time)
measured value $(mon.value_www_in)
average $(mon.average_www_in) pm $(mon.stddev_www_in)"
showstate => { "incoming.www" };
This produces the following standard output:
R: State of otherprocs peaked at Tue Dec 1 12:12:21 2014
R: The peak measured state was q = 98:
R: Frequency: [kjournald] |** (2/98)
R: Frequency: [pdflush] |** (2/98)
R: Frequency: /var/cfengine/bin/cf-execd|** (2/98)
R: Frequency: COMMAND |* (1/98)
R: Frequency: init [5] |* (1/98)
R: Frequency: [kthreadd] |* (1/98)
R: Frequency: [migration/0] |* (1/98)
R: Frequency: [ksoftirqd/0] |* (1/98)
R: Frequency: [events/0] |* (1/98)
R: Frequency: [khelper] |* (1/98)
R: Frequency: [kintegrityd/0] |* (1/98)
Finally, you can quote lines from files in your data for convenience:
body common control
{
bundlesequence => { "report" };
}
###########################################################
bundle agent report
{
reports:
linux::
"/etc/passwd except $(const.n)"
printfile => pr("/etc/passwd","5");
}
######################################################################
body printfile pr(file,lines)
{
file_to_print => "$(file)";
number_of_lines => "$(lines)";
}
This produces the following output:
R: /etc/passwd except
R: at:x:25:25:Batch jobs daemon:/var/spool/atjobs:/bin/bash
R: avahi:x:103:105:User for Avahi:/var/run/avahi-daemon:/bin/false
R: beagleindex:x:104:106:User for Beagle indexing:/var/cache/beagle:/bin/bash
R: bin:x:1:1:bin:/bin:/bin/bash
R: daemon:x:2:2:Daemon:/sbin:/bin/bash
Creating custom logs
Logs can be attached to any promise. In this example, an executed shell command logs a message to the standard output. CFEngine recognizes thestdoutfilename for Standard Output, in the Unix/C standard manner:
bundle agent test
{
commands:
"/tmp/myjob",
action => logme("executor");
}
############################################
body action logme(x)
{
log_repaired => "stdout";
log_string => " -> Started the $(x) (success)";
}
In the following example, a file creation promise logs different outcomes (success or failure) to different log files:
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
vars:
"software" slist => { "/root/xyz", "/tmp/xyz" };
files:
"$(software)"
create => "true",
action => logme("$(software)");
}
#
body action logme(x)
{
log_kept => "/tmp/private_keptlog.log";
log_failed => "/tmp/private_faillog.log";
log_repaired => "/tmp/private_replog.log";
log_string => "$(sys.date) $(x) promise status";
}
This generates three different logs with the following output:
atlas$ more /tmp/private_keptlog.log
Sun Dec 6 11:58:16 2009 /tmp/xyz promise status
Sun Dec 6 11:58:43 2009 /tmp/xyz promise status
Redirecting output to logs
CFEngine interfaces with the system logging tools in different ways. Syslog
is the
default log for Unix-like systems, while the event logger
is the default on Windows.
You may choose to copy a fixed level of CFEngine's standard screen messaging to the
system logger on a per-promise basis:
body common control
{
bundlesequence => { "one" };
}
bundle agent one
{
files:
"/tmp/xyz"
create => "true",
action => log;
}
body action log
{
log_level => "inform";
}
Change detection: tripwires
Doing a change detection scan is a convergent process, but it can still detect changes and present the data in a compressed format that is often more convenient than a full-scale audit. The result is less precise, but there is a trade-off between precision and cost.
To make a change tripwire, use a files promise, as shown below:
body common control
{
bundlesequence => { "testbundle" };
}
#
bundle agent testbundle
{
files:
"/home/mark/tmp" -> "me"
changes => scan_files,
depth_search => recurse("inf");
}
# library code ...
body changes scan_files
{
report_changes => "all";
update_hashes => "true";
}
body depth_search recurse(d)
{
depth => "$(d)";
}
In CFEngine Enterprise, reports of the following form are generated when these promises are kept by the agent:
Change detected File change
Sat Dec 5 18:27:44 2013 group for /tmp/testfile changed 100 -> 0
Sat Dec 5 18:27:44 2013 /tmp/testfile
Sat Dec 5 18:20:45 2013 /tmp/testfile
These reports are generated automatically in Enterprise, and are integrated into the
web-browsable knowledge map. Community edition users must extract the data and create
these themselves.
<!--- End include: /home/jenkins/workspace/build-documentation-3.6/label/DOCUMENTATION_x86_64_linux_debian_6/documentation/guide/reporting/command-line-reports.markdown
-->
FAQ
- Enterprise Installation
- Enterprise Scalability
- Policy Distribution
- Manual Execution
- Agent Email Reports
- Policy Writing
Enterprise Installation and Configuration
What steps should I take after installing CFEngine Enterprise
There are general steps to be taken outlined in Post-Installation Configuration.
In addition to this, Enterprise 3.6 uses the local mail relay, and it is assumed that the server where CFEngine Enterprise is installed on has proper mail setup.
The default FROM email for all emails sent from the Mission Portal is currently admin@organization.com. This can be changed on the CFEngine Hub in /var/cfengine/httpd/htdocs/application/config/appsettings.php:$config['appemail']
.
Can I use an existing PostgreSQL installation
No. Although CFEngine keeps its assumptions about Postgres to a bare minimum, CFEngine should use a dedicated PostgreSQL database instance to ensure there is no conflict with an existing installation.
Do I need experience with PostgreSQL
PostgreSQL is highly configurable and you should have some in-house expertise to properly configure your database installation. The defaults are well tuned for common cases but you may find optimizations depending on your hardware and OS.
What is the system user for the CFEngine dedicated PostgreSQL database and Apache server
Starting with CFEngine 3.6 there will be a system user called cfpostgres
for running the dedicated CFEngine PostgreSQL database
installation.
Similarly there will be a cfapache
system user for the Apache web server.
What are the requirements for installing CFEngine Enterprise
General Information
Users and Permissions
- CFEngine Enterprise makes an attempt to create the local users
cfapache
andcfpostgres
, as well as groupcfapache
when installing 3.6. The server must allow creation of these users and groups.
Enterprise Scalability
Is it normal to have many cf-hub processes running
- Yes, it is expected to have ~ 50 cf-hub processes running on your hub
Policy Distribution
I have added new files in masterfiles but my remote clients are not getting updates
Check that the files you expect to be distributed have matching leaf_name
pattern. If newly bootstrapped clients get those files but existing clients don't, this is certainly the problem, because bootstrapping and failsafe operation ignore leaf_name
and copy everything.
In CFEngine 3.6 masterfiles policy framework this is configurable with
input_name_patterns
in the update_def
bundle in def.cf
. See The Policy Framework for more information.
I have updated some non policy files and changes are not distributed to clients
cf_promises_validated
gates client updates. This file is only updated on the
policy server when new policy is validated. Edits to non policy files do not
trigger an update of cf_promises_validated
. You can use a separate promise to
ensure those files are continually distributed, instead of only on policy
updates.
For details see cf_promises_validated and cfe_internal_update_policy
My policy server has changed its IP address and new bootstraps don't work!
(thanks to Dan Langille in https://groups.google.com/forum/#!topic/help-cfengine/jcdIh12_lNI)
Symptom:
After the policy server was restarted with the new IP address, clients would not connect:
error: Not authorized to trust public key of server '192.168.14.113' (trustkey = false)
error: Authentication dialogue with '192.168.14.113' failed
Bootstrapping the clients also fails:
[root@dev /var/cfengine] /var/cfengine/bin/cf-agent --bootstrap 192.168.14.113
2014-06-23T13:57:07-0400 notice: R: This autonomous node assumes the role of voluntary client
2014-06-23T13:57:07-0400 notice: R: Failed to copy policy from policy server at 192.168.14.113:/var/cfengine/masterfiles
Please check
* cf-serverd is running on 192.168.14.113
* network connectivity to 192.168.14.113 on port 5308
* masterfiles 'body server control' - in particular allowconnects, trustkeysfrom and skipverify
* masterfiles 'bundle server' -> access: -> masterfiles -> admit/deny
It is often useful to restart cf-serverd in verbose mode (cf-serverd -v) on 192.168.14.113 to diagnose connection issues.
When updating masterfiles, wait (usually 5 minutes) for files to propagate to inputs on 192.168.14.113 before retrying.
2014-06-23T13:57:07-0400 notice: R: Did not start the scheduler
2014-06-23T13:57:07-0400 error: Bootstrapping failed, no input file at '/var/cfengine/inputs/promises.cf' after bootstrap
Solution:
Assuming that 661df12c960af9afdde093e0cb339b4d
is the MD5 hostkey and 192.168.14.113
is the new IP address:
cd /var/cfengine/ppkeys && mv -i root-MD5=661df12c960af9afdde093e0cb339b4d.pub root-192.168.14.113.pub
Manual Execution
How do I run a standalone policy file
The --file
or -f
option to cf-agent
specifys the policy file. The -K
or --no-lock
flag and the -I
or --inform
options are commonly used in combination with the -f
option to ensure that
all promises are skipped because of locking and for the agent to produce
informational output like successful repairs.
cf-agent -KIf ./my_standalone_policy.cf
A standalone policy file must specify a bundlesequence
. You can avoid that requirement by using the -b BUNDLENAME
flag, see below.
Why do I get Undefined body when I try to run my policy
cf-promises -f ./large-files.cf
:
./large-files.cf:14:0: error: Undefined body tidy with type delete
./large-files.cf:16:0: error: Undefined body recurse with type depth_search
The above errors indicate that the tidy
and recurse
bodies are not able
to be found by CFEngine. This is because they are not found in the file or in
one of the files it includes. Either define the body within the same policy
file or include the file that defines the body using inputs in either body
common control or body file
control.
Example: Add stdlib via body common control
body common control
{
bundlesequence => { "file_remover" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
Example: Add stdlib via body file control Body file control allows you to build modular policy. Body file control inputs are typically relative to the policy file itself.
bundle file_remover_control
{
vars:
"inputs" slist => { "$(this.promise_dir)/$(sys.local_libdir)/stdlib.cf" };
}
body file control
{
inputs => { @(file_remover_control.inputs) };
}
This policy will work correctly whether it's included by another
policy file or not. Note the body file
control
option is new since
CFEngine 3.6, so you should not use if your policy could be seen by
3.5 or earlier CFEngine clients.
How do I run a specific bundle
A specific bundle can be activated by passing the -b
or --bundlesequence
options to cf-agent
. This may be used to activate a specific bundle within a
large policy set or to run a standalone policy that does not include a body
common control
.
cf-agent -b my_bundle
If you want to activate multiple bundles in a sequence simply separate them with commas (no spaces between).
cf-agent --bundlesequence bundle1,bundle2
How do I define a class for a single run
You can use the --define
or -D
options of cf-agent
.
cf-agent -D my_class
And if you want to define multiple, simply separate them with commas (no spaces between).
cf-agent --define my_class,my_other_class
Multiple -D
flags are not supported, you have to put all the classes in one comma-separated list.
Agent Email Reports
How do I set the email where agent reports are sent
The agent report email functionality is configured in body executor control
https://github.com/cfengine/masterfiles/blob/master/controls/cf_execd.cf. It
defaults to root@$(def.domain)
which is configured in bundle common def
https://github.com/cfengine/masterfiles/blob/master/def.cf.
For details see domain.
How do I disable agent email output
You can simply remove or comment out the settings.
In 3.6.x there is a convenience class cfengine_internal_agent_email
avaiable
in bundle common def
to switch on/off agent email.
For details see cfengine_internal_agent_email.
Mustache Templating
How can I pass a data variable to template_data?
Currently you cannot pass a data variable directly to
template_data
, instead you must use one of the data-producing functions for
example mergedata
(), readjson
(), or parsejson
(). Please see the
Functions by Return Type table for a list
of all data-producing functions.
Can I render a Mustache template into a string?
Not directly, you could render a file and read that into a string, but you would need to be cautious of CF_BUFSIZE.
How do I render a section only if a given class is defined?
In this Mustache example the word 'Enterprise' will only be rendered if the class 'enterprise' is defined.
This template should not be passed a data container; it uses the datastate()
of the CFEngine system. That's where classes.enterprise
and vars.sys.cf_version
came from.
Version: CFEngine Enterprise
How do I iterate over a list?
This template should not be passed a data container; it uses the datastate()
of the CFEngine system. That's where vars.mon.listening_tcp4_ports
came from.
{{#vars.mon.listening_tcp4_ports}}
* {{.}}
{{/vars.mon.listening_tcp4_ports}}
How do I ensure that a local user is locked?
To ensure that a local user exists but is locked (for example a service
account) simply specify policy => "locked"
.
bundle agent service_accounts
{
vars:
"users" slist => { "apache", "libuuid" };
users:
!windows::
"$(users)"
policy => "locked";
}
Policy Writing
Common questions asked about policy writing.
How do I pass a data type variable
Data type variables also known as "data containers" are passed using the same syntax as passing a list.
bundle agent example
{
vars:
# First you must have a data type variable, define it inline or read from a
# file using `readjson()`.
"data" data => parsejson('[ { "x": 1 }, { "y": 2 } ]');
methods:
"use data"
usebundle => use_data(@(data));
}
bundle agent use_data(dc)
{
vars:
# Use the data
# Get its keys, or its index
"dc_index" slist => getindices(dc);
classes:
"have_x" expression => isvariable("dc[$(dc_index)][x]");
"have_z" expression => isvariable("dc[$(dc_index)][z]");
reports:
"CFEngine version '$(sys.cf_version)'";
have_x::
"Index '$(dc_index)' has key for x";
have_z::
"Index '$(dc_index)' has key for z";
}
$ cf-agent -Kf ./example.cf -b example
R: CFEngine version '3.6.4'
R: Index '0' has key for x
R: Index '1' has key for x
Additional Resources
Use the following links to learn more about CFEngine:
Reading
Learn by reading information brought to you by CFEngine experts:
Learning CFEngine 3 by Diego Zamboni
CFEngine 3 Tutorial and Cookbook by Neil Watson, a Senior UNIX/Linux system admin and a CFEngine Champion.
CFEngine Resources by Vertical Sysadmin, Inc, a sysadmin training company and an authorized CFEngine training partner.
CFEngine Development blog Posts on configuration management best practices from the CFEngine team.
Training
Online Training An introduction to CFEngine by our founder, Mark Burgess. These video recordings explain the basic principles and syntax of the CFEngine language and suggests some examples to try out.
Demos Videos, Webinars, and Keynotes which demonstrate the key capabilities of CFE Enterprise and Community editions.
CFEngine as a powerful security tool: Three presentations by Diego Zamboni
Tools
Download CFEngine code editors.
Sign Up
On-Site Training Sign up for professional training courses that provide a better understanding of CFEngine and how it can help improve configuration management in your organization.
Contact us to get more info on training courses.
Support and Community
Support Desk
- CFEngine Enterprise Support Desk Enterprise users have access to our support desk.
Forums
Help from our CFEngine community is available to all users on our Google Groups forums:
Support for CFEngine Enterprise users Help for users who have downloaded the free version of CFEngine 3 Enterprise.
help-cfengine General help for all your CFEngine questions.
Learning Resources
Sometimes the best help is already written.
- Visit our learning resources for guides, demos, training videos, and tools.
Social Media
Stay in touch. Follow us:
The #cfengine IRC channel on the irc.freenode.org network.
If you want to learn more about how CFEngine can help you and your organization, contact us.
Contribute to CFEngine
CFEngine Github
Public Bug Tracker
- Bugs and improvement suggestions can be registered with our development team in our public bug tracker. Read the bug tracker information before you submit a bug.