The Complete Examples and tutorials
Table of Content
- Example snippets
- General examples
- Administration examples
- Common promise patterns
- Aborting execution
- Change detection
- Copy single files
- Create files and directories
- Check filesystem space
- Customize message of the day
- Set up name resolution with DNS
- Ensure a service is enabled and running
- Find the MAC address
- Install packages
- Mount NFS filesystem
- Set up time management through NTP
- Ensure a process is not running
- Restart a process
- Distribute ssh keys
- Set up sudo
- Updating from a central policy server
- Measuring examples
- Software administration examples
- Commands, scripts, and execution examples
- File and directory examples
- Interacting with directory services
- File template examples
- Database examples
- Network examples
- System security examples
- System information examples
- System administration examples
- System file examples
- Windows registry examples
- File permissions
- User management examples
- Tutorials
- High availability
- JSON and YAML support in CFEngine
- Installing CFEngine Enterprise agent
- Managing local users
- Managing network time protocol
- Package management
- Managing processes and services
- Writing CFEngine policy
- Distributing files from a central location
- File editing
- Reporting and remediation of security vulnerabilities
- Masterfiles Policy Framework upgrade
- Tags for variables, classes, and bundles
- Custom inventory
- Dashboard alerts
- Integrating alerts with PagerDuty
- Integrating alerts with ticketing systems
- Integrating with Sumo Logic
- Rendering files with Mustache templates
- Reporting
- File comparison
- Writing and serving policy
Links to examples
- Example snippets: This section is divided into topical areas and includes many examples of policy and promises. Each of the snippets can be easily copied or downloaded to a policy server and used as is.
Note: CFEngine also includes a small set of examples by default, which can be found in /var/cfengine/share/doc/examples
.
See also:
Tutorial for running examples
In this tutorial, you will perform the following:
- Create a simple "Hello World!" example policy file
- Make the example a standalone policy
- Make the example an executable script
- Add the example to the main policy file (
promises.cf
)
Note if your CFEngine administrator has enabled continuous deployment of the policy from a Version control System, your changes may be overwritten!
"Hello world" policy example
Policies contain bundles, which are collections of promises. A promise is a declaration of intent. Bundles allow related promises to be grouped together, as illustrated in the steps that follow.
Following these steps, you will login to your policy server via the SSH protocol, use the vi command line editor to create a policy file named hello_world.cf, and create a bundle that calls a promise to display some text.
- Log into a running server machine using ssh (PuTTY may be used if using Windows).
- Type
sudo su
for super user (enter your password if prompted). - To get to the masterfiles directory, type
cd /var/cfengine/masterfiles
. - Create the file with the command:
vi hello_world.cf
- In the vi editor, enter
i
for "Insert" and enter the following content (ie. copy and paste from a text editor):cf3 [file=hello_world.cf] bundle agent hello_world { reports: any:: "Hello World!"; }
- Exit the "Insert" mode by pressing the "esc" button. This will return to the command prompt.
- Save the changes to the file by typing
:w
then "Enter". - Exit vi by typing
:q
then "Enter".
In the policy file above, we have defined an agent bundle named hello_world
. Agent
bundles are only evaluated by cf-agent, the agent component of CFEngine.
This bundle promises to report on any class of hosts.
Activate a bundle manually
Activate the bundle manually by executing the following command at prompt:
/var/cfengine/bin/cf-agent --no-lock --file ./hello_world.cf --bundlesequence hello_world
This command instructs CFEngine to ignore locks, load
the hello_world.cf
policy, and activate the hello_world
bundle. See the output below:
/var/cfengine/bin/cf-agent --no-lock --file ./hello_world.cf --bundlesequence hello_world
2013-08-20T14:03:43-0500 notice: R: Hello World!
As you get familiar with CFEngine, you'll probably start shortening this command to this equivalent:
/var/cfengine/bin/cf-agent -Kf ./hello_world.cf -b hello_world
Note the full path to the binary in the above command. CFEngine stores its binaries in /var/cfengine/bin
on Linux and Unix systems. Your path might vary depending on your platform and the packages your are using.
CFEngine uses /var because it is one of the Unix file systems that resides locally.
Thus, CFEngine can function even if everything else fails
(your other file systems, your network, and even system binaries) and possibly repair problems.
Make the example stand alone
Instead of specifying the bundle sequence on the command line (as it was above), a body common
control section can be added to
the policy file. The body common control refers to those promises that are hard-coded into
all CFEngine components and therefore affect the behavior of all components. Note that only
one body common control
is allowed per agent activation.
Go back into vi by typing "vi" at the prompt. Then type i
to insert
body common control to hello_world.cf
. Place it above bundle agent hello_world, as
shown in the following example:
body common control
{
bundlesequence => { "hello_world" };
}
bundle agent hello_world
{
reports:
any::
"Hello World!";
}
Now press "esc" to exit the "Insert" mode, then type :w
to save the file changes and "Enter".
Exit vi by typing :q
then "Enter." This will return to the prompt.
Execute the following command:
/var/cfengine/bin/cf-agent --no-lock --file ./hello_world.cf
notice: R: Hello World!
Note: It may be necessary to add a reference to the standard library within the body common control section, and remove the bundlesequence
line.
Example:
body common control
{
inputs => {
"libraries/cfengine_stdlib.cf",
};
}
Make the example an executable script
Add the #!
marker ("shebang") to hello_world.cf
in order to invoke CFEngine policy as an executable script:
Again type "vi" then "Enter" then i
to insert the following:
#!/var/cfengine/bin/cf-agent --no-lock
Add it before body common control, as shown below:
#!/var/cfengine/bin/cf-agent --no-lock
body common control
{
bundlesequence => { "hello_world" };
}
bundle agent hello_world
{
reports:
any::
"Hello World!";
}
Now exit "Insert" mode by pressing "esc". Save file changes by typing :w
then "Enter"
then exit vi by typing :q
then "Enter". This will return to the prompt.
Make the policy file executable:
chmod +x ./hello_world.cf
And it can now be run directly:
./hello_world.cf
2013-08-20T14:39:34-0500 notice: R: Hello World!
Integrating the example into your main policy
Make the example policy part of your main policy by doing the following on your policy server:
Ensure the example is located in
/var/cfengine/masterfiles
.If the example contains a
body common control
section, delete it. That section will look something like this:codebody common control { bundlesequence => { "hello_world" }; }
You cannot have duplicate control bodies (i.e. two agent control bodies, one in the main file and one in the example) as CFEngine won't know which it should use and they may conflict.
To resolve this, copy the contents of the control body section from the
example into the identically named control body section in the main policy
file /var/cfengine/masterfiles/promises.cf
and then remove the control body
from the example.
Insert the example's bundle name in the
bundlesequence
section of the main policy file/var/cfengine/masterfiles/promises.cf
:codebundlesequence => { ... "hello_world", ... };
Insert the policy file name in the
inputs
section of the main policy file/var/cfengine/masterfiles/promises.cf
:codeinputs => { ... "hello_world.cf", ... };
You must also remove any inputs section from the example that includes the external library:
codeinputs => { "libraries/cfengine_stdlib.cf" };
This is necessary, since
cfengine_stdlib.cf
is already included in the inputs section of the master policy.The example policy will now be executed every five minutes along with the rest of your main policy.
Notes: You may have to fill the example with data before it will work.
For example, the LDAP query in active_directory.cf
needs a domain name.
In the variable declaration, replace "cftesting" with your domain name:
vars:
# NOTE: Edit this to your domain, e.g. "corp"
"domain_name" string => "cftesting";
Example snippets
- General examples
- Administration examples
- Measuring examples
- Software administration examples
- Commands, scripts, and execution examples
- File and directory examples
- File template examples
- Database examples
- Network examples
- System security examples
- System information examples
- System administration examples
- System file examples
- Windows registry examples
- User management
General examples
Basic example
To get started with CFEngine, you can imagine the following template for entering examples. This part of the code is common to all the examples.
body common control
{
bundlesequence => { "main" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent main
{
# example
}
The general pattern
The general pattern of the syntax is like this (colors in html version: red, CFEngine word; blue, user-defined word):
bundle component name(parameters)
{
what_type:
where_when::
## Traditional comment
"promiser" -> { "promisee1", "promisee2" },
comment => "The intention ...",
handle => "unique_id_label",
attribute_1 => body_or_value1,
attribute_2 => body_or_value2;
}
Hello world
body common control
{
bundlesequence => { "hello" };
}
bundle agent hello
{
reports:
linux::
"Hello world!";
}
Array example
body common control
{
bundlesequence => { "array" };
}
bundle common g
{
vars:
"array[1]" string => "one";
"array[2]" string => "two";
}
bundle agent array
{
vars:
"localarray[1]" string => "one";
"localarray[2]" string => "two";
reports:
linux::
"Global $(g.array[1]) and $(localarray[2])";
}
Administration examples
Ordering promises
This counts to five by default. If we change '/bin/echo one' to '/bin/echox one', then the command will fail, causing us to skip five and go to six instead.
This shows how dependencies can be chained in spite of the order of promises in the bundle.
Normally the order of promises in a bundle is followed, within each promise type, and the types are ordered according to normal ordering.
body common control
{
bundlesequence => { "order" };
}
bundle agent order
{
vars:
"list" slist => { "three", "four" };
commands:
ok_later::
"/bin/echo five";
otherthing::
"/bin/echo six";
any::
"/bin/echo one" classes => d("ok_later","otherthing");
"/bin/echo two";
"/bin/echo $(list)";
preserved_class::
"/bin/echo seven";
}
body classes d(if,else)
{
promise_repaired => { "$(if)" };
repair_failed => { "$(else)" };
persist_time => "0";
}
Aborting execution
body common control
{
bundlesequence => { "testbundle" };
version => "1.2.3";
}
body agent control
{
abortbundleclasses => { "invalid.Hr16" };
}
bundle agent testbundle
{
vars:
"userlist" slist => { "xyz", "mark", "jeang", "jonhenrik", "thomas", "eben" };
methods:
"any" usebundle => subtest("$(userlist)");
}
bundle agent subtest(user)
{
classes:
"invalid" not => regcmp("[a-z][a-z][a-z][a-z]","$(user)");
reports:
!invalid::
"User name $(user) is valid at 4 letters";
invalid::
"User name $(user) is invalid";
}
Common promise patterns
This section includes includes common promise patterns. Refer to them as you write policy for your system.
- Aborting execution
- Change detection
- Check filesystem space
- Copy single files
- Create files and directories
- Customize message of the day
- Distribute ssh keys
- Ensure a process is not running
- Ensure a service is enabled and running
- Find the MAC address
- Install packages
- Mount NFS filesystem
- Restart a process
- Set up sudo
- Set up time management through NTP
- Set up name resolution with DNS
- Updating from a central policy server
Aborting execution
Sometimes it is useful to abort a bundle execution if certain conditions are not met,
for example when validating input to a bundle. The following policy uses a list of
regular expressions for classes, or class expressions that cf-agent
will watch out for.
If any of these classes becomes defined, it will cause the current bundle to be aborted.
body common control
{
bundlesequence => { "example" };
}
body agent control
{
abortbundleclasses => { "invalid" };
}
bundle agent example
{
vars:
#"userlist" slist => { "mark", "john" }; # contains all valid entries
"userlist" slist => { "mark", "john", "thomas" }; # contains one invalid entry
classes:
"invalid" not => regcmp("[a-z][a-z][a-z][a-z]","$(userlist)"); # The class 'invalid' is set if the user name does not
# contain exactly four un-capitalized letters (bundle
# execution will be aborted if set)
reports:
!invalid::
"User name $(userlist) is valid at 4 letters";
}
This policy can be found in
/var/cfengine/share/doc/examples/abort.cf
and downloaded directly from
github.
This is how the policy runs when the userlist is valid:
cf-agent -f unit_abort.cf
R: User name mark is valid at 4 letters
R: User name john is valid at 4 letters
This is how the policy runs when the userlist contains an invalid entry:
cf-agent -f unit_abort.cf
Bundle example aborted on defined class "invalid"
To run this example file as part of your main policy you need to make an additional change:
There cannot be two body agent control
in the main policy. Delete the
body agent control
section from /var/cfengine/masterfiles/unit_abort.cf
.
Copy and paste abortbundleclasses => { "invalid" };
into
/var/cfengine/masterfiles/controls/cf_agent.cf
. If you add it to
the end of the file it should look something like this:
...
# dryrun => "true";
abortbundleclasses => { "invalid" };
}
Change detection
This policy will look for changes recursively in a directory.
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
files:
"/home/mark/tmp/web" # Directory to monitor for changes.
changes => detect_all_change,
depth_search => recurse("inf");
}
body changes detect_all_change
{
report_changes => "all";
update_hashes => "true";
}
body depth_search recurse(d)
{
depth => "$(d)";
}
This policy can be found in
/var/cfengine/share/doc/examples/change_detect.cf
and downloaded directly from
github.
Here is an example run.
First, let's create some files for CFEngine to monitor:
# mkdir /etc/example
# date > /etc/example/example.conf
CFEngine detects new files and adds them to the file integrity database:
cf-agent -f unit_change_detect.cf
2013-06-06T20:53:26-0700 error: /example/files/'/etc/example':
File '/etc/example/example.conf' was not in 'md5' database - new file found
cf-agent -f unit_change_detect.cf -K
If there are no changes, CFEngine runs silently:
cf-agent -f unit_change_detect.cf
Now let's update the mtime, and then the mtime and content. CFEngine will notice the changes and record the new profile:
# touch /etc/example/example.conf # update mtime
# cf-agent -f unit_change_detect.cf -K
2013-06-06T20:53:52-0700 error: Last modified time for
'/etc/example/example.conf' changed 'Thu Jun 6 20:53:18 2013'
-> 'Thu Jun 6 20:53:49 2013'
# date >> /etc/example/example.conf # update mtime and content
# cf-agent -f unit_change_detect.cf -K
2013-06-06T20:54:01-0700 error: Hash 'md5' for '/etc/example/example.conf' changed!
2013-06-06T20:54:01-0700 error: /example/files/'/etc/example': Updating hash for
'/etc/example/example.conf' to 'MD5=8576cb25c9f78bc9ab6afd2c32203ca1'
2013-06-06T20:54:01-0700 error: Last modified time for '/etc/example/example.conf'
changed 'Thu Jun 6 20:53:49 2013' -> 'Thu Jun 6 20:53:59 2013'
#
Copy single files
This is a standalone policy example that will copy single files,
locally (local_cp
) and from a remote site (secure_cp
).
The CFEngine Standard Library (cfengine_stdlib.cf) should be
included in the /var/cfengine/inputs/libraries/
directory and
inputs list as below.
body common control
{
bundlesequence => { "mycopy" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent mycopy
{
files:
"/tmp/test_plain"
Path and name of the file we wish to copy to
comment => "/tmp/test_plain promises to be an up-to-date copy of /bin/echo to demonstrate copying a local file",
copy_from => local_cp("$(sys.workdir)/bin/file");
Copy locally from path/filename
"/tmp/test_remote_plain"
comment => "/tmp/test_plain_remote promises to be a copy of cfengine://serverhost.example.org/repo/config-files/motd",
copy_from => secure_cp("/repo/config-files/motd", "serverhost.example.org");
}
Copy remotely from path/filename and specified host ```
This policy can be found in
/var/cfengine/share/doc/examples/copy_copbl.cf
and downloaded directly from
github.
Create files and directories
The following is a standalone policy that will create the file
/home/mark/tmp/test_plain
and the directory /home/mark/tmp/test_dir/
and set permissions on both.
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
files:
"/home/mark/tmp/test_plain"
The promiser specifies the path and name of the file.
perms => system,
create => "true";
The perms
attribute sets the file permissions as defined in the system
body below. The create
attribute makes sure that the files exists. If it
doesn't, CFEngine will create it.
"/home/mark/tmp/test_dir/."
perms => system,
create => "true";
The trailing /.
in the filename tells CFEngine that the promiser is a
directory.
}
body perms system
{
mode => "0640";
}
This body sets permissions to "0640"
This policy can be found in
/var/cfengine/share/doc/examples/create_filedir.cf
and downloaded directly from
github.
Example output:
cf-agent -f unit_create_filedir.cf -I
2013-06-08T14:56:26-0700 info: /example/files/'/home/mark/tmp/test_plain': Created file '/home/mark/tmp/test_plain', mode 0640
2013-06-08T14:56:26-0700 info: /example/files/'/home/mark/tmp/test_dir/.': Created directory '/home/mark/tmp/test_dir/.'
2013-06-08T14:56:26-0700 info: /example/files/'/home/mark/tmp/test_dir/.': Object '/home/mark/tmp/test_dir' had permission 0755, changed it to 0750
Check filesystem space
Check how much space (in KB) is available on a directory's current partition.
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
classes:
"has_space" expression => isgreaterthan($(free), 0);
vars:
"free" int => diskfree("/tmp");
reports:
has_space::
"The filesystem has free space";
!has_space::
"The filesystem has NO free space";
}
R: The filesystem has free space
This policy can be found in
/var/cfengine/share/doc/examples/diskfree.cf
and downloaded directly from
github.
Example output:
cf-agent -f unit_diskfree.cf
R: Freedisk 48694692
df -k /tmp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 149911836 93602068 48694692 66% /
Customize message of the day
The Message of the Day is displayed when you log in or connect to a server. It typically shows information about the operating system, license information, last login, etc.
It is often useful to customize the Message of the Day to inform your users
about some specifics of the system they are connecting to. In this example we
render a /etc/motd
using a mustache template and add useful information as:
- The role of the server ( staging / production )
- The hostname of the server
- The CFEngine version we are running on the host
- The CFEngine role of the server ( client / hub )
- The administrative contacts details conditionally to the environment
- The primary Ipv4 IP address
- The number of packages updates available for this host
The bundle is defined like this:
bundle agent env_classification
{
vars:
# We use presence of key files to know the hosts environment
"environment_semaphores" slist => { "/etc/prod", "/etc/staging" };
"environment"
string => ifelse( filesexist( @(environment_semaphores) ), "incoherent",
fileexists("/etc/prod"), "production",
fileexists("/etc/staging"), "staging",
"unknown" );
"env_issue_msg"
string => ifelse( strcmp( "incoherent", $(environment) ),
"WARNING: Environment incoherent (multiple environment semaphores)",
strcmp( "unknown", $(environment) ),
"WARNING: Environment unknown (missing environment semaphores)",
"Host environment classified as $(environment)");
# This is to extract the cfengine role ( hub or client )
"cf_role"
string => ifelse( "policy_server", "Policy Server", "Policy Client");
classes:
# We define a class for the selected environment. It's useful for making
# decisions in other policies
"env_$(environment)"
expression => "any",
scope => "namespace";
}
bundle agent env_info
{
vars:
## Based on the environment we define different contacts.
"admin_contact"
slist => { "admin@acme.com", "oncall@acme.com" },
if => strcmp( $(env_classification.environment), "production" );
"admin_contact"
slist => { "dev@acme.com" },
if => strcmp( $(env_classification.environment), "staging" );
"admin_contact"
slist => { "root@localhost" },
if => strcmp( $(env_classification.environment), "unknown" );
## This is to extract the available package updates status
"updates_available"
data => packageupdatesmatching(".*", ".*", ".*", ".*");
"count_updates" int => length("updates_available");
classes:
# We define a class indicating there are updates available, it might be
# useful for various different policies.
"have_updates_available"
expression => isgreaterthan( $(count_updates), 0),
scope => "namespace";
}
bundle agent motd {
vars:
"motd_path" string => "/etc/motd";
# It's considered best practice to prepare a data container holding the
# information you need to render the template instead of relying on
# current datastate()
# First we extract currently defined classes from datastate(), then we
# construct the template data.
"_state" data => datastate(),
if => not( isvariable ( $(this.promiser) ) ); # Limit recursive growth
# and multiple calls to
# datastate() over
# multiple passes.
"template_data"
data => mergedata('{ "fqhost": "$(sys.fqhost)",
"primary_ip": "$(sys.ipv4)",
"cf_version": "$(sys.cf_version)",
"issue_msg": "$(env_classification.env_issue_msg)",
"cf_role": "$(env_classification.cf_role)",
"count_updates": "$(env_info.count_updates)",
"contacts": env_info.admin_contact,
"classes": _state[classes]
}');
files:
"$(motd_path)"
create => "true",
template_method => "inline_mustache",
template_data => @(template_data),
edit_template_string => '# Managed by CFEngine
{{{issue_msg}}}
***
*** Welcome to {{{fqhost}}}
***
* *** * CFEngine Role: {{{cf_role}}}
* *** * CFEngine Version:{{{cf_version}}}
* *** *
* * Host IP: {{{primary_ip}}}
*** {{#classes.have_updates_available}}{{{count_updates}}} package updates available.{{/classes.have_updates_available}}{{^classes.have_updates_available}}No package updates available or status unknown.{{/classes.have_updates_available}}
* *
* *
* *
For support contact:{{#contacts}}
- {{{.}}}{{/contacts}}$(const.n)';
}
bundle agent __main__
{
methods:
"env_classification";
"env_info";
"motd";
}
This policy can be found in
/var/cfengine/share/doc/examples/mustache_template_motd.cf
and downloaded directly from
github.
Example run:
cf-agent -KIf ./mustache_template_motd.cf; cat /etc/motd
info: Updated rendering of '/etc/motd' from mustache template 'inline'
info: files promise '/etc/motd' repaired
# Managed by CFEngine
WARNING: Environment unknown (missing environment semaphores)
***
*** Welcome to nickanderson-thinkpad-w550s
***
* *** * CFEngine Role: Policy Client
* *** * CFEngine Version:3.17.0
* *** *
* * Host IP: 192.168.42.189
*** No package updates available or status unknown.
* *
* *
* *
For support contact:
- root@localhost
Set up name resolution with DNS
There are many ways to configure name resolution. A simple and straightforward approach is to implement this as a simple editing promise for the /etc/resolv.conf
file.
body common control
{
bundlesequence => { "edit_name_resolution" };
}
bundle agent edit_name_resolution
{
files:
"/tmp/resolv.conf" # This is for testing, change to "$(sys.resolv)" to put in production
comment => "Add lines to the resolver configuration",
create => "true", # Make sure the file exists, create it if not
edit_line => resolver, # Call the resolver bundle defined below to do the editing
edit_defaults => empty; # Baseline memory model of file to empty before processing
# bundle edit_line resolver
}
bundle edit_line resolver
{
insert_lines:
any::
# Class/context where you use the below nameservers. Change to appropriate class
# for your system (if not any::, for example server_group::, ubuntu::, etc.)
# insert the search domain or name servers we want
"search mydomain.tld"
location => start; # Replace mydomain.tld with your domain name
# The search line will always be at the start of the file
"nameserver 128.39.89.8";
"nameserver 128.39.74.66";
}
body edit_defaults empty
{
empty_file_before_editing => "true";
}
body location start
{
before_after => "before";
}
Example run:
# cf-agent -f unit_edit_name_resolution.cf # set up resolv.conf
# cat /tmp/resolv.conf # show resolv.conf
search mydomain.tld
nameserver 128.39.89.8
nameserver 128.39.74.66
# echo 'nameserver 0.0.0.0' >> /tmp/resolv.conf # mess up resolv.conf
# cf-agent -f ./unit_edit_name_resolution.cf -KI # heal resolv.conf
2013-06-08T18:38:12-0700 info: This agent is bootstrapped to '192.168.183.208'
2013-06-08T18:38:12-0700 info: Running full policy integrity checks
2013-06-08T18:38:12-0700 info: /edit_name_resolution/files/'/tmp/resolv.conf': Edit file '/tmp/resolv.conf'
# cat /tmp/resolv.conf # show healed resolv.conf
search mydomain.tld
nameserver 128.39.89.8
nameserver 128.39.74.66
#
Ensure a service is enabled and running
This example shows how to ensure services are started or stopped appropriately.
body file control
{
inputs => { "$(sys.libdir)/services.cf", "$(sys.libdir)/commands.cf" };
}
bundle agent main
{
vars:
linux::
"enable[ssh]"
string => ifelse( "debian|ubuntu", "ssh", "sshd"),
comment => "The name of the ssh service varies on different platforms.
Here we set the name of the service based on existing
classes and defaulting to `sshd`";
"disable[apache]"
string => ifelse( "debian|ubuntu", "apache2", "httpd" ),
comment => "The name of the apache web service varies on different
platforms. Here we set the name of the service based on
existing classes and defaulting to `httpd`";
"enable[cron]"
string => ifelse( "debian|ubuntu", "cron", "crond" ),
comment => "The name of the cron service varies on different
platforms. Here we set the name of the service based on
existing classes and defaulting to `crond`";
"disable[cups]"
string => "cups",
comment => "Printing services are not needed on most hosts.";
"enabled" slist => getvalues( enable );
"disabled" slist => getvalues( disable );
services:
linux::
"$(enabled)" -> { "SysOps" }
service_policy => "start",
comment => "These services should be running because x, y or z.";
"$(disabled)" -> { "SysOps" }
service_policy => "stop",
comment => "These services should not be running because x, y or z.";
systemd::
"sysstat"
service_policy => "stop",
comment => "Standard service handling for sysstat only works with
systemd. Other inits need cron entries managed.";
}
This policy can be found in
/var/cfengine/share/doc/examples/services.cf
and downloaded directly from
github.
Note: Not all services behave in the standard way. Some services may require custom handling. For example it is not uncommon for some services to not provide correct return codes for status checks.
See also:
Example usage on systemd
We can see that before the policy run sysstat
is inactive, apache2
is
active, cups
is active, ssh
is active and cron
is inactive.
systemctl is-active sysstat apache2 cups ssh cron
inactive
active
active
active
inactive
Now we run the policy to converge the system to the desired state.
cf-agent --no-lock --inform --file ./services.cf
info: Executing 'no timeout' ... '/bin/systemctl --no-ask-password --global --system -q stop apache2'
info: Completed execution of '/bin/systemctl --no-ask-password --global --system -q stop apache2'
info: Executing 'no timeout' ... '/bin/systemctl --no-ask-password --global --system -q stop cups'
info: Completed execution of '/bin/systemctl --no-ask-password --global --system -q stop cups'
info: Executing 'no timeout' ... '/bin/systemctl --no-ask-password --global --system -q start cron'
info: Completed execution of '/bin/systemctl --no-ask-password --global --system -q start cron'
After the policy run we can see that systat
, apache2
, and cups
are
inactive. ssh
and cron
are active as specified in the policy.
systemctl is-active sysstat apache2 cups ssh cron
inactive
inactive
inactive
active
active
Example usage with System V
We can see that before the policy run sysstat
is not reporting status
correctly , httpd
is running, cups
is running, sshd
is running and
crond
is not running.
service sysstat status; echo $?
3
service httpd status; echo $?
httpd (pid 3740) is running...
0
service cups status; echo $?
cupsd (pid 3762) is running...
0
service sshd status; echo $?
openssh-daemon (pid 3794) is running...
0
service crond status; echo $?
crond is stopped
3
Now we run the policy to converge the system to the desired state.
cf-agent -KIf ./services.cf
info: Executing 'no timeout' ... '/etc/init.d/crond start'
info: Completed execution of '/etc/init.d/crond start'
info: Executing 'no timeout' ... '/etc/init.d/httpd stop'
info: Completed execution of '/etc/init.d/httpd stop'
info: Executing 'no timeout' ... '/etc/init.d/cups stop'
info: Completed execution of '/etc/init.d/cups stop'
After the policy run we can see that systat
is still not reporting status correctly (some services do not respond to standard checks), apache2
, and cups
are
inactive. ssh
and cron
are active as specified in the policy.
service sysstat status; echo $?
3
service httpd status; echo $?
httpd is stopped
3
service cups status; echo $?
cups is stopped
3
service sshd status; echo $?
openssh-daemon (pid 3794) is running...
0
service crond status; echo $?
crond (pid 3929) is running...
0
Find the MAC address
Finding the ethernet address can vary between operating systems.
We will use CFEngine's built in function execresult
to execute commands
adapted for different operating systems, assign the output to variables,
and filter for the MAC address. We then report on the result.
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
vars:
linux::
"interface"
string => execresult("/sbin/ifconfig eth0", "noshell");
solaris::
"interface"
string => execresult("/usr/sbin/ifconfig bge0", "noshell");
freebsd::
"interface"
string => execresult("/sbin/ifconfig le0", "noshell");
darwin::
"interface"
string => execresult("/sbin/ifconfig en0", "noshell");
# Use the CFEngine function 'regextract' to match the MAC address,
# assign it to an array called mac and set a class to indicate positive match
classes:
linux::
"ok"
expression => regextract(
".*HWaddr ([^\s]+).*(\n.*)*", # pattern to match
"$(interface)", # string to scan for pattern
"mac" # put the text that matches the pattern into this array
);
solaris|freebsd::
"ok"
expression => regextract(
".*ether ([^\s]+).*(\n.*)*",
"$(interface)",
"mac"
);
darwin::
"ok"
expression => regextract(
"(?s).*ether ([^\s]+).*(\n.*)*",
"$(interface)",
"mac"
);
# Report on the result
reports:
ok::
"MAC address is $(mac[1])"; # return first element in array "mac"
}
This policy can be found in /var/cfengine/masterfiles/example_find_mac_addr.cf
Example run:
cf-agent -f example_find_mac_addr.cf
2013-06-08T16:59:19-0700 notice: R: MAC address is a4:ba:db:d7:59:32
While the above illustrates the flexiblity of CFEngine in running external commands and parsing their output, as of CFEngine 3.3.0, Nova 2.2.0 (2011), you can get the MAC address natively:
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
vars:
linux::
"interface" string => "eth0";
solaris::
"interface" string => "bge0";
freebsd::
"interface" string => "le0";
darwin::
"interface" string => "en0";
reports:
"MAC address of $(interface) is: $(sys.hardware_mac[$(interface)])";
}
Install packages
Install desired packages.
body common control
{
bundlesequence => { "install_packages" };
inputs => { "libraries/cfengine_stdlib.cf" };
}
bundle agent install_packages
{
vars:
"desired_packages"
slist => { # list of packages we want
"ntp",
"lynx"
};
packages:
"$(desired_packages)" # operate on listed packages
package_policy => "add", # What to do with packages: install them.
package_method => generic; # Infer package manager (e.g. apt, yum) from the OS.
}
Caution: package management is a dirty business. If things don't go smoothly
using the generic method, you may have to use a method specific to your package
manager and get to your elbows in the details. But try generic
first. You
may get lucky.
Mind package names can differ OS to OS. For example, Apache httpd is "httpd" on Red Hat, and "apache2" on Debian.
Version comparison can be tricky when involving multipart version identifiers with numbers and letters.
CFEngine downloads the necessary packages from the default repositories if they are not present on the local machine, then installs them if they are not already installed.
Example run:
dpkg -r lynx ntp # remove packages so CFEngine has something to repair
(Reading database ... 234887 files and directories currently installed.)
Removing lynx ...
Removing ntp ...
* Stopping NTP server ntpd [ OK ]
Processing triggers for ureadahead ...
Processing triggers for man-db ...
cf-agent -f install_packages.cf # install packages
dpkg -l lynx ntp # show installed packages
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================-====================-====================================================================
ii lynx 2.8.8dev.12-2ubuntu0 all Text-mode WWW Browser (transitional package)
ii ntp 1:4.2.6.p3+dfsg-1ubu amd64 Network Time Protocol daemon and utility programs
There are examples in /var/cfengine/share/doc/examples/
of installing packages using specific package managers:
- Red Hat (unit_package_yum.cf)
- Debian (unit_package_apt.cf)
- MSI for Windows (unit_package_msi_file.cf)
- Solaris (unit_package_solaris.cf)
- SuSE Linux (unit_package_zypper.cf)
Mount NFS filesystem
Mounting an NFS filesystem is straightforward using CFEngine's storage promises. The following bundle specifies the name of a remote file system server, the path of the remote file system and the mount point directory on the local machine:
body common control
{
bundlesequence => { "mounts" };
}
bundle agent mounts
{
storage:
"/mnt" mount => nfs("fileserver","/home"); # "/mnt" is the local moint point
# "fileserver" is the remote fileserver
# "/home" is the path to the remote file system
}
body mount nfs(server,source)
{
mount_type => "nfs"; # Protocol type of remote file system
mount_source => "$(source)"; # Path of remote file system
mount_server => "$(server)"; # Name or IP of remote file system server
mount_options => { "rw" }; # List of option strings to add to the file system table ("fstab")
edit_fstab => "true"; # True/false add or remove entries to the file system table ("fstab")
}
This policy can be found in /var/cfengine/share/doc/examples/example_mount_nfs.cf
Here is an example run. At start, the filesystem is not in /etc/fstab and is not mounted:
# grep mnt /etc/fstab # filesystem is not in /etc/fstab
# df |grep mnt # filesystem is not mounted
Now we run CFEngine to mount the filesystem and add it to /etc/fstab:
cf-agent -f example_mount_nfs.cf
2013-06-08T17:48:42-0700 error: Attempting abort because mount went into a retry loop.
grep mnt /etc/fstab
fileserver:/home /mnt nfs rw
df |grep mnt
fileserver:/home 149912064 94414848 47882240 67% /mnt
Note: CFEngine errors out after it mounts the filesystem and updates /etc/fstab. There is a ticket https://cfengine.com/dev/issues/2937 open on this issue.
Set up time management through NTP
The following sets up a local NTP server that synchronizes with pool.ntp.org and clients that synchronize with your local NTP server. See bottom of this example if you don't want to build a server, but use a "brute force" method (repeated ntpdate syncs).
This example demonstrates you can have a lot of low-level detailed control if you want it.
bundle agent system_time_ntp
{
vars:
linux::
"cache_dir"
string => "$(sys.workdir)/cache"; # Cache directory for NTP config files
"ntp_conf"
string => "/etc/ntp.conf"; # Target file for NTP configuration
"ntp_server"
string => "172.16.12.161";
"ntp_network"
string => "172.16.12.0"; # IP address and netmask of your local NTP server
"ntp_mask"
string => "255.255.255.0";
"ntp_pkgs"
slist => { "ntp" }; # NTP packages to be installed to ensure service
# Define a class for the NTP server
classes:
any::
"ntp_hosts"
or => { classmatch(canonify("ipv4_$(ntp_server)")) };
# Ensure that the NTP packages are installed
packages:
ubuntu::
"$(ntp_pkgs)"
comment => "setup NTP",
package_policy => "add",
package_method => generic;
# Ensure existence of file and directory for NTP drift learning statistics
files:
linux::
"/var/lib/ntp/ntp.drift"
comment => "Enable ntp service",
create => "true";
"/var/log/ntpstats/."
comment => "Create a statistic directory",
perms => mog("644","ntp","ntp"),
create => "true";
ntp_hosts::
# Build the cache configuration file for the NTP server
"/var/cfengine/cache/ntp.conf"
comment => "Build $(this.promiser) cache file for NTP server",
create => "true",
edit_defaults => empty,
edit_line => restore_ntp_master("$(ntp_network)","$(ntp_mask)");
centos.ntp_hosts::
# Copy the cached configuration file to its target destination
"$(ntp_conf)"
comment => "Ensure $(this.promiser) in a perfect condition",
copy_from => local_cp("$(cache_dir)/ntp.conf"),
classes => if_repaired("refresh_ntpd_centos");
ubuntu.ntp_hosts::
"$(ntp_conf)"
comment => "Ensure $(this.promiser) in a perfect condition",
copy_from => local_cp("$(cache_dir)/ntp.conf"),
classes => if_repaired("refresh_ntpd_ubuntu");
!ntp_hosts::
# Build the cache configuration file for the NTP client
"$(cache_dir)/ntp.conf"
comment => "Build $(this.promiser) cache file for NTP client",
create => "true",
edit_defaults => empty,
edit_line => restore_ntp_client("$(ntp_server)");
centos.!ntp_hosts::
# Copy the cached configuration file to its target destination
"$(ntp_conf)"
comment => "Ensure $(this.promiser) in a perfect condition",
copy_from => local_cp("$(cache_dir)/ntp.conf"),
classes => if_repaired("refresh_ntpd_centos");
ubuntu.!ntp_hosts::
"$(ntp_conf)"
comment => "Ensure $(this.promiser) in a perfect condition",
copy_from => local_cp("$(cache_dir)/ntp.conf"),
classes => if_repaired("refresh_ntpd_ubuntu");
# Set classes (conditions) for to restart the NTP daemon if there have been any changes to configuration
processes:
centos::
"ntpd.*"
restart_class => "refresh_ntpd_centos";
ubuntu::
"ntpd.*"
restart_class => "refresh_ntpd_ubuntu";
# Restart the NTP daemon if the configuration has changed
commands:
refresh_ntpd_centos::
"/etc/init.d/ntpd restart";
refresh_ntpd_ubuntu::
"/etc/init.d/ntp restart";
}
#######################################################
bundle edit_line restore_ntp_master(network,mask)
{
vars:
"list"
string => "######################################
# ntp.conf-master
driftfile /var/lib/ntp/ntp.drift
statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# Consider changing the below servers to a location near you for better time
# e.g. server 0.europe.pool.ntp.org, or server 0.no.pool.ntp.org etc.
server 0.centos.pool.ntp.org
server 1.centos.pool.ntp.org
server 2.centos.pool.ntp.org
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict -4 default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1
# Hosts on local network are less restricted.
restrict $(network) mask $(mask) nomodify notrap";
insert_lines:
"$(list)";
}
#######################################################
bundle edit_line restore_ntp_client(serverip)
{
vars:
"list"
string => "######################################
# This file is protected by cfengine #
######################################
# ntp.conf-client
driftfile /var/lib/ntp/ntp.drift
statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict -4 default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1
server $(serverip)
restrict $(serverip) nomodify";
insert_lines:
"$(list)";
}
This policy can be found in /var/cfengine/share/doc/examples/example_ntp.cf
If you don't want to build a server, you might do like this:
bundle agent time_management
{
vars:
any::
"ntp_server"
string => "no.pool.ntp.org";
commands:
any::
"/usr/sbin/ntpdate $(ntp_server)"
contain => silent;
}
This is a hard reset of the time, it corrects it immediately. This may cause problems if there are large deviations in time and you are using time sensitive software on your system. An NTP daemon setup as shown above, on the other hand, slowly adapts the time to avoid causing disruption. In addition, the NTP daemon can be configured to learn your system's time drift and automatically adjust for it without having to be in touch with the server at all times.
Ensure a process is not running
This is a standalone policy that will kill the sleep
process. You can adapt
it to make sure that any undesired process is not running.
body common control
{
bundlesequence => { "process_kill" };
}
bundle agent process_kill
{
processes:
"sleep"
signals => { "term", "kill" }; #Signals are presented as an ordered list to the process.
#On Windows, only the kill signal is supported, which terminates the process.
}
This policy can be found in /var/cfengine/share/doc/examples/unit_process_kill.cf
.
Example run:
/bin/sleep 1000 &
[1] 5370
cf-agent -f unit_process_kill.cf
[1]+ Terminated /bin/sleep 1000
Now let's do it again with inform mode turned on, and CFEngine will show the process table entry that matched the pattern we specified ("sleep"):
/bin/sleep 1000 &
[1] 5377
cf-agent -f unit_process_kill.cf -IK
2013-06-08T16:30:06-0700 info: This agent is bootstrapped to '192.168.183.208'
2013-06-08T16:30:06-0700 info: Running full policy integrity checks
2013-06-08T16:30:06-0700 info: /process_kill/processes/'sleep': Signalled 'term' (15) to process 5377 (root 5377 3854 5377 0.0 0.0 11352 0 612 1 16:30 00:00:00 /bin/sleep 1000)
[1]+ Terminated /bin/sleep 1000
If we add the -v switch to turn on verbose mode, we see the /bin/ps command CFEngine used to dump the process table:
cf-agent -f unit_process_kill.cf -Kv
...
2013-06-08T16:38:20-0700 verbose: Observe process table with /bin/ps -eo user,pid,ppid,pgid,pcpu,pmem,vsz,ni,rss,nlwp,stime,time,args
2013-06-08T16:38:20-0700 verbose: Matched 'root 5474 3854 5474 0.0 0.0 11352 0 612 1 16:38 00:00:00 /bin/sleep 1000'
...
Restart a process
This is a standalone policy that will restart three CFEngine processes if they are not running.
body common control
{
bundlesequence => { "process_restart" };
}
bundle agent process_restart
{
vars:
"component" slist => { # List of processes to monitor
"cf-monitord",
"cf-serverd",
"cf-execd"
};
processes:
"$(component)"
# Set the class "<component>_not_running" if it is not running:
restart_class => canonify("$(component)_not_running");
commands:
"/var/cfengine/bin/$(component)"
if => canonify("$(component)_not_running");
}
Notes: The canonify
function translates illegal characters to underscores, e.g. start_cf-monitord
becomes start_cf_monitord
. Only alphanumerics and underscores are allowed in CFEngine identifiers (names of variables, classes, bundles, etc.)
This policy can be found in /var/cfengine/share/doc/examples/unit_process_restart.cf
.
Example run:
# ps -ef |grep cf-
root 4305 1 0 15:14 ? 00:00:02 /var/cfengine/bin/cf-execd
root 4311 1 0 15:14 ? 00:00:05 /var/cfengine/bin/cf-serverd
root 4397 1 0 15:15 ? 00:00:06 /var/cfengine/bin/cf-monitord
# kill 4311
# ps -ef |grep cf-
root 4305 1 0 15:14 ? 00:00:02 /var/cfengine/bin/cf-execd
root 4397 1 0 15:15 ? 00:00:06 /var/cfengine/bin/cf-monitord
# cf-agent -f unit_process_restart.cf
# ps -ef |grep cf-
root 4305 1 0 15:14 ? 00:00:02 /var/cfengine/bin/cf-execd
root 4397 1 0 15:15 ? 00:00:06 /var/cfengine/bin/cf-monitord
root 8008 1 0 18:18 ? 00:00:00 /var/cfengine/bin/cf-serverd
#
And again, in Inform mode:
kill 8008
cf-agent -f unit_process_restart.cf -I
2013-06-08T18:19:51-0700 info: This agent is bootstrapped to '192.168.183.208'
2013-06-08T18:19:51-0700 info: Running full policy integrity checks
2013-06-08T18:19:51-0700 info: /process_restart/processes/'$(component)': Making a one-time restart promise for 'cf-serverd'
2013-06-08T18:19:51-0700 info: Executing 'no timeout' ... '/var/cfengine/bin/cf-serverd'
2013-06-08T18:19:52-0700 info: Completed execution of '/var/cfengine/bin/cf-serverd'
Distribute ssh keys
This example shows a simple ssh key distribution implementation.
The policy was designed to work with the services_autorun
feature in
the Masterfiles Policy Framework. The
services_autorun
feature can be enabled from the augments_file. If
you do not have a def.json
in the root of your masterfiles directory
simply create it with the following content.
{
"classes": {
"services_autorun": [ "any" ]
}
}
In the following example we will manage the authorized_keys
file for
bob
, frank
, and kelly
.
For each listed user the ssh_key_distribution
bundle is activated if
the user exists on the system. Once activated the
ssh_key_distribution
bundle ensures that proper permissions are set
on the users .ssh
directory (home is assumed to be in
/home/username
) and ensures that the users .ssh/authorized_keys
is
a copy of the users authorized_keys
file as found on the server as
defined in the ssh_key_info
bundle.
Let's assume we collected all users' public keys into a single directory on the server and that users exist on the clients (and have corresponding home directory).
Note: special variable $(sys.policy_hub)
contains the hostname of
the policy server.
To deploy this policy simply place it in the services/autorun
directory of your masterfiles.
body common control
{
bundlesequence => { "autorun_ssh_key_distribution" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle common ssh_key_info
{
meta:
"description"
string => "This bundle defines common ssh key information, like which
directory and server keys should be sourced from.";
vars:
"key_server" string => "$(sys.policy_hub)";
# We set the path to the repo in a common bundle so that we can reference
# the same path when defining access rules and when copying files.
# This directory is expected to contain one file for each users authorized
# keys, named for the username. For example: /srv/ssh_authorized_keys/kelly
"repo_path" string => "/srv/ssh_authorized_keys";
}
bundle agent autorun_ssh_key_distribution
{
meta:
# Here we simply tag the bundle for use with the `services_autorun`
# feature.
"tags" slist => { "autorun" };
vars:
"users" slist => { "bob", "frank", "kelly" };
methods:
"Distribute SSH Keys"
usebundle => ssh_key_distribution( $(users) ),
if => userexists( $(users) ),
comment => "It's important that we make sure each of these users
ssh_authorized_keys file has the correct content and
permissions so that they can successfully log in, if
the user exists on the executing agents host.";
}
bundle agent ssh_key_distribution(users)
{
meta:
"description"
string => "Ensure that specified users are able to log in using their ssh
keys";
vars:
# We get the users UID so that we can set permission appropriately
"uid[$(users)]" int => getuid( $(users) );
files:
"/home/$(users)/.ssh/."
create => "true",
perms => mo( 700, "$(uid[$(users)])"),
comment => "It is important to set the proper restrictive permissions and
ownership so that the ssh authorized_keys feature works
correctly.";
"/home/$(users)/.ssh/authorized_keys"
perms => mo( 600, "$(uid[$(users)])" ),
copy_from => remote_dcp( "$(ssh_key_info.repo_path)/$(users)",
$(ssh_key_info.key_server) ),
comment => "We centrally manage and users authorized keys. We source each
users complete authorized_keys file from the central server.";
}
bundle server ssh_key_access_rules
{
meta:
"description"
string => "This bundle handles sharing the directory where ssh keys
are distributed from.";
access:
# Only hosts with class `policy_server` should share the path to ssh
# authorized_keys
policy_server::
"$(ssh_key_info.repo_path)"
admit => { @(def.acl) },
comment => "We share the ssh authorized keys with all authorized
hosts.";
}
This policy can be found in
/var/cfengine/share/doc/examples/simple_ssh_key_distribution.cf
and downloaded directly from
github.
Example Run:
First make sure the users exist on your system.
root@host001:~# useradd bob
root@host001:~# useradd frank
root@host001:~# useradd kelly
Then update the policy and run it:
cf-agent -Kf update.cf; cf-agent -KI
info: Installing cfe_internal_non_existing_package...
info: Created directory '/home/bob/.ssh/.'
info: Owner of '/home/bob/.ssh' was 0, setting to 1002
info: Object '/home/bob/.ssh' had permission 0755, changed it to 0700
info: Copying from '192.168.56.2:/srv/ssh_authorized_keys/bob'
info: Owner of '/home/bob/.ssh/authorized_keys' was 0, setting to 1002
info: Created directory '/home/frank/.ssh/.'
info: Owner of '/home/frank/.ssh' was 0, setting to 1003
info: Object '/home/frank/.ssh' had permission 0755, changed it to 0700
info: Copying from '192.168.56.2:/srv/ssh_authorized_keys/frank'
info: Owner of '/home/frank/.ssh/authorized_keys' was 0, setting to 1003
info: Created directory '/home/kelly/.ssh/.'
info: Owner of '/home/kelly/.ssh' was 0, setting to 1004
info: Object '/home/kelly/.ssh' had permission 0755, changed it to 0700
info: Copying from '192.168.56.2:/srv/ssh_authorized_keys/kelly'
info: Owner of '/home/kelly/.ssh/authorized_keys' was 0, setting to 1004
Set up sudo
Setting up sudo is straightforward, we recommend managing it by copying trusted files from a repository. The following bundle will copy a master sudoers file to /etc/sudoers
(/tmp/sudoers
in this example - change it to /etc/sudoers
to use in production).
body common control
{
bundlesequence => { "sudoers" };
inputs => { "libraries/cfengine_stdlib.cf" };
}
bundle agent sudoers
{
# Define the master location of the sudoers file
vars:
"master_location" string => "/var/cfengine/masterfiles";
# Copy the master sudoers file to /etc/sudoers
files:
"/tmp/sudoers" # change to /etc/sudoers to use in production
comment => "Make sure the sudo configuration is secure and up to date",
perms => mog("440","root","root"),
copy_from => secure_cp("$(master_location)/sudoers","$(sys.policy_hub)");
}
We recommend editing the master sudoers file using visudo
or a similar tool. It is possible to use CFEngine's file editing capabilities to edit sudoers directly, but this does not guarantee syntax correctness and you might end up locked out.
Example run:
cf-agent -f temp.cf -KI
2013-06-08T19:13:21-0700 info: This agent is bootstrapped to '192.168.183.208'
2013-06-08T19:13:22-0700 info: Running full policy integrity checks
2013-06-08T19:13:23-0700 info: Copying from '192.168.183.208:/var/cfengine/masterfiles/sudoers'
2013-06-08T19:13:23-0700 info: /sudoers/files/'/tmp/sudoers': Object '/tmp/sudoers' had permission 0600, changed it to 0440
For reference we include an example of a simple sudoers file:
# /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
Defaults env_reset
# User privilege specification
root ALL=(ALL) ALL
# Allow members of group sudo to execute any command after they have
# provided their password
%sudo ALL=(ALL) ALL
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL
john ALL=(ALL) ALL
Updating from a central policy server
This is a conceptual example without any test policy associated with it.
The default policy shipped with CFEngine contains a centralized updating of policy that covers more subtleties than this example, and handles fault tolerance. Here is the main idea behind it. For simplicity, we assume that all hosts are on network 10.20.30.* and that the central policy server is 10.20.30.123.
bundle agent update
{
vars:
"master_location" string => "/var/cfengine/masterfiles";
"policy_server" string => "10.20.30.123";
comment => "IP address to locate your policy host.";
files:
"$(sys.workdir)/inputs"
perms => system("600"),
copy_from => remote_cp("$(master_location)",$(policy_server)),
depth_search => recurse("inf"); # This ensures recursive copying of all subdirectories
"$(sys.workdir)/bin"
perms => system("700"),
copy_from => remote_cp("/usr/local/sbin","localhost"),
depth_search => recurse("inf"); # This ensures recursive copying of all subdirectories
}
In addition the server needs to grant access to the clients, this is done in the body server control
:
body server control
{
allowconnects => { "127.0.0.1" , "10.20.30.0/24" };
allowallconnects => { "127.0.0.1" , "10.20.30.0/24" };
trustkeysfrom => { "127.0.0.1" , "10.20.30.0/24" };
}
Since we assume that all hosts are on network 10.20.30.* they will be granted access. In the default policy this is set to $(sys.policy_hub)/16
, i.e. all hosts in the same class B network as the hub will gain access. You will need to modify the access control list in body server control
if you have clients outside of the policy server's class B network.
Granting access to files and folders needs to be done using access
type promises in a server
bundle, for example, bundle server my_access_rules()
:
bundle server my_access_rules()
{
access:
10_20_30_123::
"/var/cfengine/masterfiles"
admit => { "127.0.0.1", "10.20.30.0/24" };
}
Measuring examples
Measurements
body common control
{
bundlesequence => { "report" };
}
body monitor control
{
forgetrate => "0.7";
histograms => "true";
}
bundle agent report
{
reports:
"
Free memory read at $(mon.av_free_memory_watch)
cf_monitord read $(mon.value_monitor_self_watch)
";
}
bundle monitor watch
{
measurements:
# Test 1 - extract string matching
"/home/mark/tmp/testmeasure"
handle => "blonk_watch",
stream_type => "file",
data_type => "string",
history_type => "weekly",
units => "blonks",
match_value => find_blonks,
action => sample_min("10");
# Test 2 - follow a special process over time
# using cfengine's process cache to avoid resampling
"/var/cfengine/state/cf_rootprocs"
handle => "monitor_self_watch",
stream_type => "file",
data_type => "int",
history_type => "static",
units => "kB",
match_value => proc_value(".*cf-monitord.*",
"root\s+[0-9.]+\s+[0-9.]+\s+[0-9.]+\s+[0-9.]+\s+([0-9]+).*");
# Test 3, discover disk device information
"/bin/df"
handle => "free_disk_watch",
stream_type => "pipe",
data_type => "slist",
history_type => "static",
units => "device",
match_value => file_system;
# Update this as often as possible
# Test 4
"/tmp/file"
handle => "line_counter",
stream_type => "file",
data_type => "counter",
match_value => scanlines("MYLINE.*"),
history_type => "log";
}
body match_value scanlines(x)
{
select_line_matching => "^$(x)$";
}
body action sample_min(x)
{
ifelapsed => "$(x)";
expireafter => "$(x)";
}
body match_value find_blonks
{
select_line_number => "2";
extraction_regex => "Blonk blonk ([blonk]+).*";
}
body match_value free_memory # not willy!
{
select_line_matching => "MemFree:.*";
extraction_regex => "MemFree:\s+([0-9]+).*";
}
body match_value proc_value(x,y)
{
select_line_matching => "$(x)";
extraction_regex => "$(y)";
}
body match_value file_system
{
select_line_matching => "/.*";
extraction_regex => "(.*)";
}
Software administration examples
- Software and patch installation
- Postfix mail configuration
- Set up a web server
- Add software packages to the system
- Application baseline
- Service management (windows)
- Software distribution
- Web server modules
- Ensure a service is enabled and running
- Managing Software
- Install packages
Software and patch installation
Example for Debian:
body common control
{
bundlesequence => { "packages" };
}
body agent control
{
environment => { "DEBIAN_FRONTEND=noninteractive" };
}
bundle agent packages
{
vars:
# Test the simplest case -- leave everything to the yum smart manager
"match_package" slist => {
"apache2"
# "apache2-mod_php5",
# "apache2-prefork",
# "php5"
};
packages:
"$(match_package)"
package_policy => "add",
package_method => apt;
}
body package_method apt
{
any::
# ii acpi 0.09-3ubuntu1
package_changes => "bulk";
package_list_command => "/usr/bin/dpkg -l";
package_list_name_regex => "ii\s+([^\s]+).*";
package_list_version_regex => "ii\s+[^\s]+\s+([^\s]+).*";
# package_list_arch_regex => "none";
package_installed_regex => ".*"; # all reported are installed
#package_name_convention => "$(name)_$(version)_$(arch)";
package_name_convention => "$(name)";
# Use these only if not using a separate version/arch string
# package_version_regex => "";
# package_name_regex => "";
# package_arch_regex => "";
package_add_command => "/usr/bin/apt-get --yes install";
package_delete_command => "/usr/bin/apt-get --yes remove";
package_update_command => "/usr/bin/apt-get --yes dist-upgrade";
#package_verify_command => "/bin/rpm -V";
}
Examples MSI for Windows, by name:
body common control
{
bundlesequence => { "packages" };
}
bundle agent packages
{
vars:
"match_package" slist => {
"7zip"
};
packages:
"$(match_package)"
package_policy => "update",
package_select => ">=",
package_architectures => { "x86_64" },
package_version => "3.00",
package_method => msi_vmatch;
}
body package_method msi_vmatch
{
package_changes => "individual";
package_file_repositories => { "$(sys.workdir)\software_updates\windows", "s:\su" };
package_installed_regex => ".*";
package_name_convention => "$(name)-$(version)-$(arch).msi";
package_add_command => "\"$(sys.winsysdir)\msiexec.exe\" /qn /i";
package_update_command => "\"$(sys.winsysdir)\msiexec.exe\" /qn /i";
package_delete_command => "\"$(sys.winsysdir)\msiexec.exe\" /qn /x";
}
Windows MSI by version:
body common control
{
bundlesequence => { "packages" };
}
bundle agent packages
{
vars:
"match_package" slist => {
"7zip"
};
packages:
"$(match_package)"
package_policy => "update",
package_select => ">=",
package_architectures => { "x86_64" },
package_version => "3.00",
package_method => msi_vmatch;
}
body package_method msi_vmatch
{
package_changes => "individual";
package_file_repositories => { "$(sys.workdir)\software_updates\windows", "s:\su" };
package_installed_regex => ".*";
package_name_convention => "$(name)-$(version)-$(arch).msi";
package_add_command => "\"$(sys.winsysdir)\msiexec.exe\" /qn /i";
package_update_command => "\"$(sys.winsysdir)\msiexec.exe\" /qn /i";
package_delete_command => "\"$(sys.winsysdir)\msiexec.exe\" /qn /x";
}
Examples for solaris:
bundle agent example_using_ips_package_method
{
packages:
solaris::
"shell/zsh"
package_policy => "add",
package_method => ips;
}
bundle agent example_using_solaris_package_method
{
files:
solaris::
"/tmp/$(admin_file)"
create => "true",
edit_defaults => empty_file, # defined in stdlib
edit_line => create_solaris_admin_file; # defined in stdlib
packages:
solaris::
"SMCzlib"
package_policy => "add",
package_method => solaris( "SMCzlib",
"zlib-1.2.3-sol10-sparc-local",
"$(admin_file)");
}
bundle agent example_using_solaris_install_package_method
{
packages:
solaris::
"SMCzlib"
package_method => solaris_install("/tmp/SMCzlib.adminfile")
}
bundle agent example_using_pkgsrc_module
{
packages:
solaris::
"vim"
policy => "present",
package_module => pkgsrc;
}
Examples for yum based systems:
body common control
{
bundlesequence => { "packages" };
inputs => { "cfengine_stdlib.cf" };
}
bundle agent packages
{
vars:
# Test the simplest case -- leave everything to the yum smart manager
"match_package" slist => {
"apache2",
"apache2-mod_php5",
"apache2-prefork",
"php5"
};
packages:
"$(match_package)"
package_policy => "add",
package_method => yum;
}
SuSE Linux's package manager zypper is the most powerful alternative:
body common control
{
bundlesequence => { "packages" };
inputs => { "cfengine_stdlib.cf" }
}
bundle agent packages
{
vars:
# Test the simplest case -- leave everything to the zypper smart manager
"match_package" slist => {
"apache2",
"apache2-mod_php5",
"apache2-prefork",
"php5"
};
packages:
"$(match_package)"
package_policy => "add",
package_method => zypper;
}
Postfix mail configuration
body common control
{
inputs => { "$(sys.libdir)/stdlib.cf" };
bundlesequence => { postfix };
}
bundle agent postfix
{
vars:
"prefix" string => "/etc";
"smtpserver" string => "localhost";
"mailrelay" string => "mailx.example.org";
files:
"$(prefix)/main.cf"
edit_line => prefix_postfix;
"$(prefix)/sasl-passwd"
create => "true",
perms => mo("0600","root"),
edit_line => append_if_no_line("$(smtpserver) _$(sys.fqhost):chmsxrcynz4etfrejizhs22");
}
bundle edit_line prefix_postfix
{
#
# Value have the form NAME = "quoted space separated list"
#
vars:
"ps[relayhost]" string => "[$(postfix.mailrelay)]:587";
"ps[mydomain]" string => "iu.hio.no";
"ps[smtp_sasl_auth_enable]" string => "yes";
"ps[smtp_sasl_password_maps]" string => "hash:/etc/postfix/sasl-passwd";
"ps[smtp_sasl_security_options]" string => "";
"ps[smtp_use_tls]" string => "yes";
"ps[default_privs]" string => "mailman";
"ps[inet_protocols]" string => "all";
"ps[inet_interfaces]" string => "127.0.0.1";
"parameter_name" slist => getindices("ps");
delete_lines:
"$(parameter_name).*";
insert_lines:
"$(parameter_name) = $(ps[$(parameter_name)])";
}
bundle edit_line AppendIfNSL(parameter)
{
insert_lines:
"$(parameter)"; # This is default
}
Set up a web server
Adapt this template to your operating system by adding multiple classes. Each web server runs something like the present module, which is entered into the bundlesequence like this:
bundle agent web_server(state)
{
vars:
"document_root" string => "/";
####################################################
# Site specific configuration - put it in this file
####################################################
"site_http_conf" string => "/home/mark/CFEngine-inputs/httpd.conf";
####################################################
# Software base
####################################################
"match_package" slist => {
"apache2",
"apache2-mod_php5",
"apache2-prefork",
"php5"
};
#########################################################
processes:
web_ok.on::
"apache2"
restart_class => "start_apache";
off::
"apache2"
process_stop => "/etc/init.d/apache2 stop";
#########################################################
commands:
start_apache::
"/etc/init.d/apache2 start"; # or startssl
#########################################################
packages:
"$(match_package)"
package_policy => "add",
package_method => zypper,
classes => if_ok("software_ok");
#########################################################
files:
software_ok::
"/etc/sysconfig/apache2"
edit_line => fixapache,
classes => if_ok("web_ok");
#########################################################
reports:
!software_ok.on::
"The web server software could not be installed";
#########################################################
classes:
"on" expression => strcmp("$(state)","on");
"off" expression => strcmp("$(state)","off");
}
bundle edit_line fixapache
{
vars:
"add_modules" slist => {
"ssl",
"php5"
};
"del_modules" slist => {
"php3",
"php4",
"jk"
};
insert_lines:
"APACHE_CONF_INCLUDE_FILES=\"$(web_server.site_http_conf)\"";
field_edits:
#####################################################################
# APACHE_MODULES="actions alias ssl php5 dav_svn authz_default jk" etc..
#####################################################################
"APACHE_MODULES=.*"
# Insert module "columns" between the quoted RHS
# using space separators
edit_field => quotedvar("$(add_modules)","append");
"APACHE_MODULES=.*"
# Delete module "columns" between the quoted RHS
# using space separators
edit_field => quotedvar("$(del_modules)","delete");
# if this line already exists, edit it
}
Add software packages to the system
body common control
{
inputs => { "$(sys.libdir)/packages.cf" }
bundlesequence => { "packages" };
}
bundle agent packages
{
vars:
"match_package" slist => {
"apache2",
"apache2-mod_php5",
"apache2-prefork",
"php5"
};
packages:
solaris::
"$(match_package)"
package_policy => "add",
package_method => solaris;
redhat|SuSE::
"$(match_package)"
package_policy => "add",
package_method => yum_rpm;
methods:
# equivalent in 3.6, no OS choices
"" usebundle => ensure_present($(match_package));
}
Note you can also arrange to hide all the differences between package managers on an OS basis, but since some OSs have multiple managers, this might not be 100 percent correct.
Application baseline
bundle agent app_baseline
{
methods:
windows::
"any" usebundle => detect_adobereader;
}
bundle agent detect_adobereader
{
vars:
windows::
"value1" string => registryvalue("HKEY_LOCAL_MACHINE\SOFTWARE\Adobe\Acrobat Reader\9.0\Installer", "ENU_GUID");
"value2" string => registryvalue("HKEY_LOCAL_MACHINE\SOFTWARE\Adobe\Acrobat Reader\9.0\Installer", "VersionMax");
"value3" string => registryvalue("HKEY_LOCAL_MACHINE\SOFTWARE\Adobe\Acrobat Reader\9.0\Installer", "VersionMin");
classes:
windows::
"is_correct" and => {
strcmp($(value1), "{AC76BA86-7AD7-1033-7B44-A93000000001}"),
strcmp($(value2), "90003"),
islessthan($(value3), "10001" )
};
reports:
windows.!is_correct::
'Adobe Reader is not correctly deployed - got "$(value1)", "$(value2)", "$(value3)"';
}
Service management (windows)
body common control
{
bundlesequence => { "winservice" };
}
bundle agent winservice
{
vars:
"bad_services" slist => { "Alerter", "ClipSrv" };
services:
windows::
"$(bad_services)"
service_policy => "disable",
comment => "Disable services that create security issues";
}
Software distribution
bundle agent check_software
{
vars:
# software to install if not installed
"include_software" slist => {
"7-zip-4.50-$(sys.arch).msi"
};
# this software gets updated if it is installed
"autoupdate_software" slist => {
"7-zip"
};
# software to uninstall if it is installed
"exclude_software" slist => {
"7-zip-4.65-$(sys.arch).msi"
};
methods:
# "any" usebundle => add_software( "@(check_software.include_software)", "$(sys.policy_hub)" );
# "any" usebundle => update_software( "@(check_software.autoupdate_software)", "$(sys.policy_hub)" );
# "any" usebundle => remove_software( "@(check_software.exclude_software)", "$(sys.policy_hub)" );
}
bundle agent add_software(pkg_name)
{
vars:
# dir to install from locally - can also check multiple directories
"local_software_dir" string => "C:\Program Files\Cfengine\software\add";
files:
"$(local_software_dir)"
copy_from => remote_cp("/var/cfengine/master_software_updates/$(sys.flavour)_$(sys.arch)/add", "$(srv)"),
depth_search => recurse("1"),
classes => if_repaired("got_newpkg"),
comment => "Copy software from remote repository";
packages:
# When to check if the package is installed ?
got_newpkg|any::
"$(pkg_name)"
package_policy => "add",
package_method => msi_implicit( "$(local_software_dir)" ),
classes => if_else("add_success", "add_fail" ),
comment => "Install new software, if not already present";
reports::
add_fail::
"Failed to install one or more packages";
}
#########################################################################
bundle agent update_software(sw_names)
{
vars:
# dir to install from locally - can also check multiple directories
"local_software_dir" string => "C:\Program Files\Cfengine\software\update";
files:
"$(local_software_dir)"
copy_from => remote_cp("/var/cfengine/master_software_updates/$(sys.flavour)_$(sys.arch)/update", "$(srv)"),
depth_search => recurse("1"),
classes => if_repaired("got_newpkg"),
comment => "Copy software updates from remote repository";
packages:
# When to check if the package is updated ?
got_newpkg|any::
"$(sw_names)"
package_policy => "update",
package_select => ">=", # picks the newest update available
package_architectures => { "$(sys.arch)" }, # install 32 or 64 bit package ?
package_version => "1.0", # at least version 1.0
package_method => msi_explicit( "$(local_software_dir)" ),
classes => if_else("update_success", "update_fail");
reports:
update_fail::
"Failed to update one or more packages";
}
#########################################################################
bundle agent remove_software(pkg_name)
{
vars:
# dir to install from locally - can also check multiple directories
"local_software_dir" string => "C:\Program Files\Cfengine\software\remove";
files:
"$(local_software_dir)"
copy_from => remote_cp("/var/cfengine/master_software_updates/$(sys.flavour)_$(sys.arch)/remove", "$(srv)"),
depth_search => recurse("1"),
classes => if_repaired("got_newpkg"),
comment => "Copy removable software from remote repository";
packages:
got_newpkg::
"$(pkg_name)"
package_policy => "delete",
package_method => msi_implicit( "$(local_software_dir)" ),
classes => if_else("remove_success", "remove_fail" ),
comment => "Remove software, if present";
reports::
remove_fail::
"Failed to remove one or more packages";
}
Web server modules
The problem of editing the correct modules into the list of standard modules for the Apache web server. This example is based on the standard configuration deployment of SuSE Linux. Simply provide the list of modules you want and another list that you don't want.
body common control
{
inputs => { "$(sys.libdir)/stdlib.cf" };
bundlesequence => {
apache
};
}
bundle agent apache
{
files:
SuSE::
"/etc/sysconfig/apache2"
edit_line => fixapache;
}
bundle edit_line fixapache
{
vars:
"add_modules" slist => {
"dav",
"dav_fs",
"ssl",
"php5",
"dav_svn",
"xyz",
"superduper"
};
"del_modules" slist => {
"php3",
"jk",
"userdir",
"imagemap",
"alias"
};
insert_lines:
"APACHE_CONF_INCLUDE_FILES=\"/site/masterfiles/local-http.conf\"";
field_edits:
#####################################################################
# APACHE_MODULES="authz_host actions alias ..."
#####################################################################
# Values have the form NAME = "quoted space separated list"
"APACHE_MODULES=.*"
# Insert module "columns" between the quoted RHS
# using space separators
edit_field => quoted_var($(add_modules), "append");
"APACHE_MODULES=.*"
# Delete module "columns" between the quoted RHS
# using space separators
edit_field => quoted_var($(del_modules), "delete");
# if this line already exists, edit it
}
Commands, scripts, and execution examples
- Command or script execution
- Change directory for command
- Commands example
- Execresult example
- Methods
- Method validation
- Trigger classes
Command or script execution
Execute a command, for instance to start a MySQL service. Note that simple shell commands like rm or mkdir cannot be managed by CFEngine, so none of the protections that CFEngine offers can be applied to the process. Moreover, this starts a new process, adding to the burden on the system.
body common control
{
bundlesequence => { "my_commands" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent my_commands
{
commands:
Sunday.Hr04.Min05_10.myhost::
"/usr/bin/update_db";
any::
"/etc/mysql/start"
contain => setuid("mysql");
}
Change directory for command
body common control
{
bundlesequence => { "example" };
}
body contain cd(dir)
{
chdir => "${dir}";
useshell => "true";
}
bundle agent example
{
commands:
"/bin/pwd"
contain => cd("/tmp");
}
Commands example
body common control
{
bundlesequence => { "my_commands" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent my_commands
{
commands:
Sunday.Hr04.Min05_10.myhost::
"/usr/bin/update_db";
any::
"/etc/mysql/start"
contain => setuid("mysql");
}
Execresult example
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
vars:
"my_result" string => execresult("/bin/ls /tmp","noshell");
reports:
"Variable is $(my_result)";
}
Methods
body common control
{
bundlesequence => { "testbundle" };
version => "1.2.3";
}
bundle agent testbundle
{
vars:
"userlist" slist => { "mark", "jeang", "jonhenrik", "thomas", "eben" };
methods:
"any" usebundle => subtest("$(userlist)");
}
bundle agent subtest(user)
{
commands:
"/bin/echo Fix $(user)";
reports:
"Finished doing stuff for $(user)";
}
Method validation
body common control
{
bundlesequence => { "testbundle" };
version => "1.2.3";
}
body agent control
{
abortbundleclasses => { "invalid" };
}
bundle agent testbundle
{
vars:
"userlist" slist => { "xyz", "mark", "jeang", "jonhenrik", "thomas", "eben" };
methods:
"any" usebundle => subtest("$(userlist)");
}
bundle agent subtest(user)
{
classes:
"invalid" not => regcmp("[a-z][a-z][a-z][a-z]","$(user)");
reports:
!invalid::
"User name $(user) is valid at 4 letters";
invalid::
"User name $(user) is invalid";
}
Trigger classes
body common control
{
any::
bundlesequence => { "insert" };
}
bundle agent insert
{
vars:
"v" string => "
One potato
Two potato
Three potahto
Four
";
files:
"/tmp/test_insert"
edit_line => Insert("$(insert.v)"),
edit_defaults => empty,
classes => trigger("edited");
commands:
edited::
"/bin/echo make bananas";
reports:
edited::
"The potatoes are bananas";
}
bundle edit_line Insert(name)
{
insert_lines:
"Begin$(const.n) $(name)$(const.n)End";
}
body edit_defaults empty
{
empty_file_before_editing => "true";
}
body classes trigger(x)
{
promise_repaired => { $(x) };
}
File and directory examples
- Create files and directories
- Copy single files
- Copy directory trees
- Disabling and rotating files
- Add lines to a file
- Check file or directory permissions
- Commenting lines in a file
- Copy files
- Copy and flatten directory
- Copy then edit a file convergently
- Deleting lines from a file
- Deleting lines exception
- Delete files recursively
- Editing files
- Editing tabular files
- Inserting lines in a file
- Back references in filenames
- Add variable definitions to a file
- Linking files
- Listing files-pattern in a directory
- Locate and transform files
- BSD flags
- Search and replace text
- Selecting a region in a file
- Warn if matching line in file
Create files and directories
Create files and directories and set permissions.
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/home/mark/tmp/test_plain"
perms => system,
create => "true";
"/home/mark/tmp/test_dir/."
perms => system,
create => "true";
}
body perms system
{
mode => "0640";
}
Copy single files
Copy single files, locally (local_cp) or from a remote site (secure_cp). The Community Open Promise-Body Library (COPBL; cfengine_stdlib.cf) should be included in the /var/cfengine/inputs/ directory and input as below.
body common control
{
bundlesequence => { "mycopy" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent mycopy
{
files:
"/home/mark/tmp/test_plain"
copy_from => local_cp("$(sys.workdir)/bin/file");
"/home/mark/tmp/test_remote_plain"
copy_from => secure_cp("$(sys.workdir)/bin/file","serverhost");
}
Copy directory trees
Copy directory trees, locally (local_cp) or from a remote site (secure_cp). (depth_search => recurse("")) defines the number of sublevels to include, ("inf") gets entire tree.
body common control
{
bundlesequence => { "my_recursive_copy" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent my_recursive_copy
{
files:
"/home/mark/tmp/test_dir"
copy_from => local_cp("$(sys.workdir)/bin/."),
depth_search => recurse("inf");
"/home/mark/tmp/test_dir"
copy_from => secure_cp("$(sys.workdir)/bin","serverhost"),
depth_search => recurse("inf");
}
Disabling and rotating files
Use the following simple steps to disable and rotate files. See the Community Open Promise-Body Library if you wish more details on what disable and rotate does.
body common control
{
bundlesequence => { "my_disable" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent my_disable
{
files:
"/home/mark/tmp/test_create"
rename => disable;
"/home/mark/tmp/rotate_my_log"
rename => rotate("4");
}
Add lines to a file
There are numerous approaches to adding lines to a file. Often the order of a configuration file is unimportant, we just need to ensure settings within it. A simple way of adding lines is show below.
body common control
{
any::
bundlesequence => { "insert" };
}
bundle agent insert
{
vars:
"lines" string =>
"
One potato
Two potato
Three potatoe
Four
";
files:
"/tmp/test_insert"
create => "true",
edit_line => append_if_no_line("$(insert.lines)");
}
Also you could write this using a list variable:
body common control
{
any::
bundlesequence => { "insert" };
}
bundle agent insert
{
vars:
"lines" slist => { "One potato", "Two potato",
"Three potatoe", "Four" };
files:
"/tmp/test_insert"
create => "true",
edit_line => append_if_no_line("@(insert.lines)");
}
Check file or directory permissions
bundle agent check_perms
{
vars:
"ns_files" slist => {
"/local/iu/logs/admin",
"/local/iu/logs/security",
"/local/iu/logs/updates",
"/local/iu/logs/xfer"
};
files:
NameServers::
"/local/dns/pz"
perms => mo("644","dns"),
depth_search => recurse("1"),
file_select => exclude("secret_file");
"/local/iu/dns/pz/FixSerial"
perms => m("755"),
file_select => plain;
"$(ns_files)"
perms => mo("644","dns"),
file_select => plain;
"$(ftp)/pub"
perms => mog("644","root","other");
"$(ftp)/pub"
perms => m("644"),
depth_search => recurse("inf");
"$(ftp)/etc" perms => mog("111","root","other");
"$(ftp)/usr/bin/ls" perms => mog("111","root","other");
"$(ftp)/dev" perms => mog("555","root","other");
"$(ftp)/usr" perms => mog("555","root","other");
}
Commenting lines in a file
body common control
{
version => "1.2.3";
inputs => { "$(sys.libdir)/stdlib.cf" };
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/home/mark/tmp/cf3_test"
create => "true",
edit_line => myedit("second");
}
bundle edit_line myedit(parameter)
{
vars:
"edit_variable" string => "private edit variable is $(parameter)";
replace_patterns:
# replace shell comments with C comments
"#(.*)"
replace_with => C_comment,
select_region => MySection("New section");
}
body replace_with C_comment
{
replace_value => "/* $(match.1) */"; # backreference 0
occurrences => "all"; # first, last all
}
body select_region MySection(x)
{
select_start => "\[$(x)\]";
select_end => "\[.*\]";
}
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/home/mark/tmp/comment_test"
create => "true",
edit_line => comment_lines_matching;
}
bundle edit_line comment_lines_matching
{
vars:
"regexes" slist => { "one.*", "two.*", "four.*" };
replace_patterns:
"^($(regexes))$"
replace_with => comment("# ");
}
body replace_with comment(c)
{
replace_value => "$(c) $(match.1)";
occurrences => "all";
}
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/home/mark/tmp/comment_test"
create => "true",
edit_line => uncomment_lines_matching("\s*mark.*","#");
}
bundle edit_line uncomment_lines_matching(regex,comment)
{
replace_patterns:
"#($(regex))$" replace_with => uncomment;
}
body replace_with uncomment
{
replace_value => "$(match.1)";
occurrences => "all";
}
Copy files
files:
"/var/cfengine/inputs"
handle => "update_policy",
perms => m("600"),
copy_from => u_scp("$(master_location)",@(policy_server)),
depth_search => recurse("inf"),
file_select => input_files,
action => immediate;
"/var/cfengine/bin"
perms => m("700"),
copy_from => u_scp("/usr/local/sbin","localhost"),
depth_search => recurse("inf"),
file_select => cf3_files,
action => immediate,
classes => on_change("reload");
Copy and flatten directory
body common control
{
bundlesequence => { "testbundle" };
version => "1.2.3";
}
bundle agent testbundle
{
files:
"/home/mark/tmp/testflatcopy"
comment => "test copy promise",
copy_from => mycopy("/home/mark/LapTop/words","127.0.0.1"),
perms => system,
depth_search => recurse("inf"),
classes => satisfied("copy_ok");
"/home/mark/tmp/testcopy/single_file"
comment => "test copy promise",
copy_from => mycopy("/home/mark/LapTop/Cfengine3/trunk/README","127.0.0.1"),
perms => system;
reports:
copy_ok::
"Files were copied..";
}
body perms system
{
mode => "0644";
}
body depth_search recurse(d)
{
depth => "$(d)";
}
body copy_from mycopy(from,server)
{
source => "$(from)";
servers => { "$(server)" };
compare => "digest";
verify => "true";
copy_backup => "true"; #/false/timestamp
purge => "false";
type_check => "true";
force_ipv4 => "true";
trustkey => "true";
collapse_destination_dir => "true";
}
body classes satisfied(x)
{
promise_repaired => { "$(x)" };
persist_time => "0";
}
body server control
{
allowconnects => { "127.0.0.1" , "::1" };
allowallconnects => { "127.0.0.1" , "::1" };
trustkeysfrom => { "127.0.0.1" , "::1" };
}
bundle server my_access_rules()
{
access:
"/home/mark/LapTop"
admit => { "127.0.0.1" };
}
Copy then edit a file convergently
To convergently chain a copy followed by edit, you need a staging file. First you copy to the staging file. Then you edit the final file and insert the staging file into it as part of the editing. This is convergent with respect to both stages of the process.
bundle agent master
{
files:
"$(final_destination)"
create => "true",
edit_line => fix_file("$(staging_file)"),
edit_defaults => empty,
perms => mo("644","root"),
action => ifelapsed("60");
}
bundle edit_line fix_file(f)
{
insert_lines:
"$(f)"
# insert this into an empty file to reconstruct
insert_type => "file";
replace_patterns:
"searchstring"
replace_with => With("replacestring");
}
Deleting lines from a file
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
files:
"/tmp/resolv.conf" # test on "/tmp/resolv.conf" #
create => "true",
edit_line => resolver,
edit_defaults => def;
}
bundle edit_line resolver
{
vars:
"search" slist => { "search iu.hio.no cfengine.com", "nameserver 128.39.89.10" };
delete_lines:
"search.*";
insert_lines:
"$(search)" location => end;
}
body edit_defaults def
{
empty_file_before_editing => "false";
edit_backup => "false";
max_file_size => "100000";
}
body location start
{
# If not line to match, applies to whole text body
before_after => "before";
}
body location end
{
# If not line to match, applies to whole text body
before_after => "after";
}
Deleting lines exception
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/tmp/passwd_excerpt"
create => "true",
edit_line => MarkNRoot;
}
bundle edit_line MarkNRoot
{
delete_lines:
"mark.*|root.*" not_matching => "true";
}
Delete files recursively
The rm_rf and rm_rf_depth bundles in the standard library make it easy to prune directory trees.
Editing files
This is a huge topic. See also See Add lines to a file, See Editing tabular files, etc. Editing a file can be complex or simple, depending on needs.
Here is an example of how to comment out lines matching a number of patterns:
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent testbundle
{
vars:
"patterns" slist => { "finger.*", "echo.*", "exec.*", "rstat.*",
"uucp.*", "talk.*" };
files:
"/etc/inetd.conf"
edit_line => comment_lines_matching("@(testbundle.patterns)","#");
}
Editing tabular files
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
vars:
"userset" slist => { "one-x", "two-x", "three-x" };
files:
# Make a copy of the password file
"/home/mark/tmp/passwd"
create => "true",
edit_line => SetUserParam("mark","6","/set/this/shell");
"/home/mark/tmp/group"
create => "true",
edit_line => AppendUserParam("root","4","@(userset)");
commands:
"/bin/echo" args => $(userset);
}
bundle edit_line SetUserParam(user,field,val)
{
field_edits:
"$(user):.*"
# Set field of the file to parameter
edit_field => col(":","$(field)","$(val)","set");
}
bundle edit_line AppendUserParam(user,field,allusers)
{
vars:
"val" slist => { @(allusers) };
field_edits:
"$(user):.*"
# Set field of the file to parameter
edit_field => col(":","$(field)","$(val)","alphanum");
}
body edit_field col(split,col,newval,method)
{
field_separator => $(split);
select_field => $(col);
value_separator => ",";
field_value => $(newval);
field_operation => $(method);
extend_fields => "true";
}
Inserting lines in a file
body common control
{
any::
bundlesequence => { "insert" };
}
bundle agent insert
{
vars:
"v" string => " One potato";
files:
"/tmp/test_insert"
create => "true",
edit_line => Insert("$(insert.v)");
}
bundle edit_line Insert(name)
{
insert_lines:
" $(name)"
whitespace_policy => { "ignore_leading", "ignore_embedded" };
}
body edit_defaults empty
{
empty_file_before_editing => "true";
}
body common control
{
any::
bundlesequence => { "insert" };
}
bundle agent insert
{
vars:
"v" string => "
One potato
Two potato
Three potatoe
Four
";
files:
"/tmp/test_insert"
create => "true",
edit_line => Insert("$(insert.v)"),
edit_defaults => empty;
}
bundle edit_line Insert(name)
{
insert_lines:
"Begin$(const.n)$(name)$(const.n)End";
}
body edit_defaults empty
{
empty_file_before_editing => "false";
}
body common control
{
any::
bundlesequence => { "insert" };
}
bundle agent insert
{
vars:
"v" slist => {
"One potato",
"Two potato",
"Three potatoe",
"Four"
};
files:
"/tmp/test_insert"
create => "true",
edit_line => Insert("@(insert.v)");
# edit_defaults => empty;
}
bundle edit_line Insert(name)
{
insert_lines:
"$(name)";
}
body edit_defaults empty
{
empty_file_before_editing => "true";
}
Back references in filenames
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
# The back reference in a path only applies to the last link
# of the pathname, so the (tmp) gets ignored
"/tmp/(cf3)_(.*)"
edit_line => myedit("second $(match.2)");
# but ...
# "/tmp/cf3_test"
# create => "true",
# edit_line => myedit("second $(match.1)");
}
bundle edit_line myedit(parameter)
{
vars:
"edit_variable" string => "private edit variable is $(parameter)";
insert_lines:
"$(edit_variable)";
}
Add variable definitions to a file
body common control
{
bundlesequence => { "setvars" };
inputs => { "cf_std_library.cf" };
}
bundle agent setvars
{
vars:
# want to set these values by the names of their array keys
"rhs[lhs1]" string => " Mary had a little pig";
"rhs[lhs2]" string => "Whose Fleece was white as snow";
"rhs[lhs3]" string => "And everywhere that Mary went";
# oops, now change pig -> lamb
files:
"/tmp/system"
create => "true",
edit_line => set_variable_values("setvars.rhs");
}
Results in:
- lhs1= Mary had a little pig
- lhs2=Whose Fleece was white as snow
- lhs3=And everywhere that Mary went
An example of this would be to add variables to /etc/sysctl.conf on Linux:
body common control
{
bundlesequence => { "setvars" };
inputs => { "cf_std_library.cf" };
}
bundle agent setvars
{
vars:
# want to set these values by the names of their array keys
"rhs[net/ipv4/tcp_syncookies]" string => "1";
"rhs[net/ipv4/icmp_echo_ignore_broadcasts]" string => "1";
"rhs[net/ipv4/ip_forward]" string => "1";
# oops, now change pig -> lamb
files:
"/etc/sysctl"
create => "true",
edit_line => set_variable_values("setvars.rhs");
}
Linking files
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
# Make a copy of the password file
"/home/mark/tmp/passwd"
link_from => linkdetails("/etc/passwd"),
move_obstructions => "true";
"/home/mark/tmp/linktest"
link_from => linkchildren("/usr/local/sbin");
#child links
}
body link_from linkdetails(tofile)
{
source => "$(tofile)";
link_type => "symlink";
when_no_source => "force"; # kill
}
body link_from linkchildren(tofile)
{
source => "$(tofile)";
link_type => "symlink";
when_no_source => "force"; # kill
link_children => "true";
when_linking_children => "if_no_such_file"; # "override_file";
}
body common control
{
any::
bundlesequence => {
"testbundle"
};
}
bundle agent testbundle
{
files:
"/home/mark/tmp/test_to" -> "someone"
depth_search => recurse("inf"),
perms => modestuff,
action => tell_me;
}
body depth_search recurse(d)
{
rmdeadlinks => "true";
depth => "$(d)";
}
body perms modestuff
{
mode => "o-w";
}
body action tell_me
{
report_level => "inform";
}
Listing files-pattern in a directory
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
vars:
"ls" slist => lsdir("/etc","p.*","true");
reports:
"ls: $(ls)";
}
Locate and transform files
body common control
{
any::
bundlesequence => {
"testbundle"
};
version => "1.2.3";
}
bundle agent testbundle
{
files:
"/home/mark/tmp/testcopy"
file_select => pdf_files,
transformer => "/usr/bin/gzip $(this.promiser)",
depth_search => recurse("inf");
}
body file_select pdf_files
{
leaf_name => { ".*.pdf" , ".*.fdf" };
file_result => "leaf_name";
}
body depth_search recurse(d)
{
depth => "$(d)";
}
BSD flags
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
files:
freebsd::
"/tmp/newfile"
create => "true",
perms => setbsd;
}
body perms setbsd
{
bsdflags => { "+uappnd","+uchg", "+uunlnk", "-nodump" };
}
Search and replace text
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/tmp/replacestring"
create => "true",
edit_line => myedit("second");
}
bundle edit_line myedit(parameter)
{
vars:
"edit_variable" string => "private edit variable is $(parameter)";
replace_patterns:
# replace shell comments with C comments
"puppet"
replace_with => With("cfengine 3");
}
body replace_with With(x)
{
replace_value => $(x);
occurrences => "first";
}
body select_region MySection(x)
{
select_start => "\[$(x)\]";
select_end => "\[.*\]";
}
Selecting a region in a file
body common control
{
version => "1.2.3";
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/tmp/testfile"
create => "true",
edit_line => myedit("second");
}
bundle edit_line myedit(parameter)
{
vars:
"edit_variable" string => "private edit variable is $(parameter)";
replace_patterns:
# comment out lines after start
"([^#].*)"
replace_with => comment,
select_region => ToEnd("Start.*");
}
body replace_with comment
{
replace_value => "# $(match.1)"; # backreference 0
occurrences => "all"; # first, last all
}
body select_region ToEnd(x)
{
select_start => $(x);
}
Warn if matching line in file
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/var/cfengine/inputs/.*"
edit_line => DeleteLinesMatching(".*cfenvd.*"),
action => WarnOnly;
}
bundle edit_line DeleteLinesMatching(regex)
{
delete_lines:
"$(regex)" action => WarnOnly;
}
body action WarnOnly
{
action_policy => "warn";
}
Interacting with directory services
Active directory example
bundle agent active_directory
{
vars:
# NOTE: Edit this to your domain, e.g. "corp", may also need more DC's after it
"domain_name" string => "cftesting";
"user_name" string => "Guest";
# NOTE: We can also extract data from remote Domain Controllers
dummy.DomainController::
"domain_controller" string => "localhost";
"userlist" slist => ldaplist(
"ldap://$(domain_controller)",
"CN=Users,DC=$(domain_name),DC=com",
"(objectClass=user)",
"sAMAccountName",
"subtree",
"none");
classes:
dummy.DomainController::
"gotuser" expression => ldaparray(
"userinfo",
"ldap://$(domain_controller)",
"CN=$(user_name),CN=Users,DC=$(domain_name),DC=com",
"(name=*)",
"subtree",
"none");
reports:
dummy.DomainController::
'Username is "$(userlist)"';
dummy.gotuser::
"Got user data; $(userinfo[name]) has logged on $(userinfo[logonCount]) times";
}
Active list users directory example
bundle agent ldap
{
vars:
"userlist" slist => ldaplist(
"ldap://cf-win2003",
"CN=Users,DC=domain,DC=cf-win2003",
"(objectClass=user)",
"sAMAccountName",
"subtree",
"none");
reports:
'Username: "$(userlist)"';
}
Active directory show users example
bundle agent ldap
{
classes:
"gotdata" expression => ldaparray(
"myarray",
"ldap://cf-win2003",
"CN=Test Pilot,CN=Users,DC=domain,DC=cf-win2003",
"(name=*)",
"subtree",
"none");
reports:
gotdata::
"Got user data";
!gotdata::
"Did not get user data";
}
LDAP interactions
body common control
{
bundlesequence => { "ldap" , "followup"};
}
bundle agent ldap
{
vars:
# Get the first matching value for "uid"
"value" string => ldapvalue("ldap://eternity.iu.hio.no","dc=cfengine,dc=com","(sn=User)","uid","subtree","none");
# Get all matching values for "uid" - should be a single record match
"list" slist => ldaplist("ldap://eternity.iu.hio.no","dc=cfengine,dc=com","(sn=User)","uid","subtree","none");
classes:
"gotdata" expression => ldaparray("myarray","ldap://eternity.iu.hio.no","dc=cfengine,dc=com","(uid=mark)","subtree","none");
"found" expression => regldap("ldap://eternity.iu.hio.no","dc=cfengine,dc=com","(sn=User)","uid","subtree","jon.*","none");
reports:
linux::
"LDAP VALUE $(value) found";
"LDAP LIST VALUE $(list)";
gotdata::
"Found specific entry data ...$(ldap.myarray[uid]),$(ldap.myarray[gecos]), etc";
found::
"Matched regex";
}
bundle agent followup
{
reports:
linux::
"Different bundle ...$(ldap.myarray[uid]),$(ldap.myarray[gecos]),...";
}
File template examples
Templating
With CFEngine you have a choice between editing deltas into files or distributing more-or-less finished templates. Which method you should choose depends should be made by whatever is easiest.
If you are managing only part of the file, and something else (e.g. a package manager) is managing most of it, then it makes sense to use CFEngine file editing.
If you are managing everything in the file, then it makes sense to make the edits by hand and install them using CFEngine. You can use variables within source text files and let CFEngine expand them locally in situ, so that you can make generic templates that apply netwide.
Example template:
MYVARIABLE = something or other
HOSTNAME = $(sys.host) # CFEngine fills this in
To copy and expand this template, you can use a pattern like this:
bundle agent get_template(final_destination,mode)
{
vars:
# This needs to ne preconfigured to your site
"masterfiles" string => "/home/mark/tmp";
"this_template" string => lastnode("$(final_destination)","/");
files:
"$(final_destination).staging"
comment => "Get template and expand variables for this host",
perms => mo("400","root"),
copy_from => remote_cp("$(masterfiles)/templates/$(this_template)","$(policy_server)"),
action => if_elapsed("60");
"$(final_destination)"
comment => "Expand the template",
create => "true",
edit_line => expand_template("$(final_destination).staging"),
edit_defaults => empty,
perms => mo("$(mode)","root"),
action => if_elapsed("60");
}
The the following driving code (based on copy then edit) can be placed in a library, after configuring to your environmental locations:
bundle agent get_template(final_destination,mode)
{
vars:
# This needs to ne preconfigured to your site
"masterfiles" string => "/home/mark/tmp";
"this_template" string => lastnode("$(final_destination)","/");
files:
"$(final_destination).staging"
comment => "Get template and expand variables for this host",
perms => mo("400","root"),
copy_from => remote_cp("$(masterfiles)/templates/$(this_template)","$(policy_server)"),
action => if_elapsed("60");
"$(final_destination)"
comment => "Expand the template",
create => "true",
edit_line => expand_template("$(final_destination).staging"),
edit_defaults => empty,
perms => mo("$(mode)","root"),
action => if_elapsed("60");
}
Database examples
Database creation
body common control
{
bundlesequence => { "dummy" };
}
body knowledge control
{
#sql_database => "postgres";
sql_owner => "postgres";
sql_passwd => ""; # No passwd
sql_type => "postgres";
}
bundle knowledge dummy
{
topics:
}
body common control
{
bundlesequence => { "databases" };
}
bundle agent databases
{
#commands:
# "/usr/bin/createdb cf_topic_maps",
# contain => as_user("mysql");
databases:
"knowledge_bank/topics"
database_operation => "create",
database_type => "sql",
database_columns => {
"topic_name,varchar,256",
"topic_comment,varchar,1024",
"topic_id,varchar,256",
"topic_type,varchar,256",
"topic_extra,varchar,26"
},
database_server => myserver;
}
body database_server myserver
{
none::
db_server_owner => "postgres";
db_server_password => "";
db_server_host => "localhost";
db_server_type => "postgres";
db_server_connection_db => "postgres";
any::
db_server_owner => "root";
db_server_password => "";
db_server_host => "localhost";
db_server_type => "mysql";
db_server_connection_db => "mysql";
}
body contain as_user(x)
{
exec_owner => "$(x)";
}
Network examples
- Find MAC address
- Client-server example
- Read from a TCP socket
- Set up a PXE boot server
- Resolver management
- Mount NFS filesystem
- Unmount NFS filesystem
- Find the MAC address
- Mount NFS filesystem
Find MAC address
Finding the ethernet address can be hard, but on Linux it is straightforward.
bundle agent test
{
vars:
linux::
"interface" string => execresult("/sbin/ifconfig eth0","noshell");
solaris::
"interface" string => execresult("/usr/sbin/ifconfig bge0","noshell");
freebsd::
"interface" string => execresult("/sbin/ifconfig le0","noshell");
darwin::
"interface" string => execresult("/sbin/ifconfig en0","noshell");
classes:
linux::
"ok" expression => regextract(
".*HWaddr ([^\s]+).*(\n.*)*",
"$(interface)",
"mac"
);
solaris::
"ok" expression => regextract(
".*ether ([^\s]+).*(\n.*)*",
"$(interface)",
"mac"
);
freebsd::
"ok" expression => regextract(
".*ether ([^\s]+).*(\n.*)*",
"$(interface)",
"mac"
);
darwin::
"ok" expression => regextract(
"(?s).*ether ([^\s]+).*(\n.*)*",
"$(interface)",
"mac"
);
reports:
ok::
"MAC address is $(mac[1])";
}
Client-server example
body common control
{
bundlesequence => { "testbundle" };
version => "1.2.3";
#fips_mode => "true";
}
bundle agent testbundle
{
files:
"/home/mark/tmp/testcopy"
comment => "test copy promise",
copy_from => mycopy("/home/mark/LapTop/words","127.0.0.1"),
perms => system,
depth_search => recurse("inf"),
classes => satisfied("copy_ok");
"/home/mark/tmp/testcopy/single_file"
comment => "test copy promise",
copy_from => mycopy("/home/mark/LapTop/Cfengine3/trunk/README","127.0.0.1"),
perms => system;
reports:
copy_ok::
"Files were copied..";
}
body perms system
{
mode => "0644";
}
body depth_search recurse(d)
{
depth => "$(d)";
}
body copy_from mycopy(from,server)
{
source => "$(from)";
servers => { "$(server)" };
compare => "digest";
encrypt => "true";
verify => "true";
copy_backup => "true"; #/false/timestamp
purge => "false";
type_check => "true";
force_ipv4 => "true";
trustkey => "true";
}
body classes satisfied(x)
{
promise_repaired => { "$(x)" };
persist_time => "0";
}
body server control
{
allowconnects => { "127.0.0.1" , "::1" };
allowallconnects => { "127.0.0.1" , "::1" };
trustkeysfrom => { "127.0.0.1" , "::1" };
# allowusers
}
bundle server my_access_rules()
{
access:
"/home/mark/LapTop"
admit => { "127.0.0.1" };
}
Read from a TCP socket
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
vars:
"my80" string => readtcp("research.iu.hio.no","80","GET /index.php HTTP/1.1$(const.r)$(const.n)Host: research.iu.hio.no$(const.r)$(const.n)$(const.r)$(const.n)",20);
classes:
"server_ok" expression => regcmp(".*200 OK.*\n.*","$(my80)");
reports:
server_ok::
"Server is alive";
!server_ok::
"Server is not responding - got $(my80)";
}
Set up a PXE boot server
Use CFEngine to set up a PXE boot server.
body common control
{
bundlesequence => { "pxe" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent pxe
{
vars:
"software" slist => {
"atftp",
"dhcp-server",
"syslinux",
"apache2"
};
"dirs" slist => {
"/tftpboot",
"/tftpboot/CFEngine/rpm",
"/tftpboot/CFEngine/inputs",
"/tftpboot/pxelinux.cfg",
"/tftpboot/kickstart",
"/srv/www/repos"
};
"tmp_location" string => "/tftpboot/CFEngine/inputs";
# Distros that we can install
"rh_distros" slist => { "4.7", "5.2" };
"centos_distros" slist => { "5.2" };
# File contents of atftp configuration
"atftpd_conf" string =>
"
ATFTPD_OPTIONS=\"--daemon \"
ATFTPD_USE_INETD=\"no\"
ATFTPD_DIRECTORY=\"/tftpboot\"
ATFTPD_BIND_ADDRESSES=\"\"
";
# File contents of DHCP configuration
"dhcpd" string =>
"
DHCPD_INTERFACE=\"eth0\"
DHCPD_RUN_CHROOTED=\"yes\"
DHCPD_CONF_INCLUDE_FILES=\"\"
DHCPD_RUN_AS=\"dhcpd\"
DHCPD_OTHER_ARGS=\"\"
DHCPD_BINARY=\"\"
";
"dhcpd_conf" string =>
"
allow booting;
allow bootp;
ddns-update-style none; ddns-updates off;
subnet 192.168.0.0 netmask 255.255.255.0 {
range 192.168.0.20 192.168.0.254;
default-lease-time 3600;
max-lease-time 4800;
option routers 192.168.0.1;
option domain-name \"test.CFEngine.com\";
option domain-name-servers 192.168.0.1;
next-server 192.168.0.1;
filename \"pxelinux.0\";
}
group {
host node1 {
# Dummy machine
hardware ethernet 00:0F:1F:94:FE:07;
fixed-address 192.168.0.11;
option host-name \"node1\";
}
host node2 {
# Dell Inspiron 1150
hardware ethernet 00:0F:1F:0E:70:E7;
fixed-address 192.168.0.12;
option host-name \"node2\";
}
}
";
# File contains of Apache2 HTTP configuration
"httpd_conf" string =>
"
<Directory /srv/www/repos>
Options Indexes
AllowOverride None
</Directory>
Alias /repos /srv/www/repos
<Directory /tftpboot/distro/RHEL/5.2>
Options Indexes
AllowOverride None
</Directory>
Alias /distro/rhel/5.2 /tftpboot/distro/RHEL/5.2
<Directory /tftpboot/distro/RHEL/4.7>
Options Indexes
AllowOverride None
</Directory>
Alias /distro/rhel/4.7 /tftpboot/distro/RHEL/4.7
<Directory /tftpboot/distro/CentOS/5.2>
Options Indexes
AllowOverride None
</Directory>
Alias /distro/centos/5.2 /tftpboot/distro/CentOS/5.2
<Directory /tftpboot/kickstart>
Options Indexes
AllowOverride None
</Directory>
Alias /kickstart /tftpboot/kickstart
<Directory /tftpboot/CFEngine>
Options Indexes
AllowOverride None
</Directory>
Alias /CFEngine /tftpboot/CFEngine
";
# File contains of Kickstart for RHEL5 configuration
"kickstart_rhel5_conf" string =>
"
auth --useshadow --enablemd5
bootloader --location=mbr
clearpart --all --initlabel
graphical
firewall --disabled
firstboot --disable
key 77244a6377a8044a
keyboard no
lang en_US
logging --level=info
url --url=http://192.168.0.1/distro/rhel/5.2
network --bootproto=dhcp --device=eth0 --onboot=on
reboot
rootpw --iscrypted $1$eOnXdDPF$279sQ//zry6rnQktkATeM0
selinux --disabled
timezone --isUtc Europe/Oslo
install
part swap --bytes-per-inode=4096 --fstype=\"swap\" --recommended
part / --bytes-per-inode=4096 --fstype=\"ext3\" --grow --size=1
%packages
@core
@base
db4-devel
openssl-devel
gcc
flex
bison
libacl-devel
libselinux-devel
pcre-devel
device-mapper-multipath
-sysreport
%post
cd /root
rpm -i http://192.168.0.1/CFEngine/rpm/CFEngine-3.0.1b1-1.el5.i386.rpm
cd /etc/yum.repos.d
wget http://192.168.0.1/repos/RHEL5.Base.repo
rpm --import /etc/pki/rpm-gpg/*
yum clean all
yum update
mkdir -p /root/CFEngine_init
cd /root/CFEngine_init
wget -nd -r http://192.168.0.1/CFEngine/inputs/
/usr/local/sbin/cf-agent -B
/usr/local/sbin/cf-agent
";
# File contains of PXElinux boot menu
"pxelinux_boot_menu" string =>
"
boot options:
rhel5 - install 32 bit i386 RHEL 5.2 (MANUAL)
rhel5w - install 32 bit i386 RHEL 5.2 (AUTO)
rhel4 - install 32 bit i386 RHEL 4.7 AS (MANUAL)
centos5 - install 32 bit i386 CentOS 5.2 (Desktop) (MANUAL)
";
# File contains of PXElinux default configuration
"pxelinux_default" string =>
"
default rhel5
timeout 300
prompt 1
display pxelinux.cfg/boot.msg
F1 pxelinux.cfg/boot.msg
label rhel5
kernel vmlinuz-RHEL5U2
append initrd=initrd-RHEL5U2 load_ramdisk=1 ramdisk_size=16384 install=http://192.168.0.1/distro/rhel/5.2
label rhel5w
kernel vmlinuz-RHEL5U2
append initrd=initrd-RHEL5U2 load_ramdisk=1 ramdisk_size=16384 ks=http://192.168.0.1/kickstart/kickstart-RHEL5U2.cfg
label rhel4
kernel vmlinuz-RHEL4U7
append initrd=initrd-RHEL4U7 load_ramdisk=1 ramdisk_size=16384 install=http://192.168.0.1/distro/rhel/4.7
label centos5
kernel vmlinuz-CentOS5.2
append initrd=initrd-CentOS5.2 load_ramdisk=1 ramdisk_size=16384 install=http://192.168.0.1/distro/centos/5.2
";
# File contains of specified PXElinux default to be a RHEL5 webserver
"pxelinux_rhel5_webserver" string =>
"
default rhel5w
label rhel5w
kernel vmlinuz-RHEL5U2
append initrd=initrd-RHEL5U2 load_ramdisk=1 ramdisk_size=16384 ks=http://192.168.0.1/kickstart/kickstart-RHEL5U2.cfg
";
# File contains of a local repository for RHEL5
"rhel5_base_repo" string =>
"
[Server]
name=Server
baseurl=http://192.168.0.1/repos/rhel5/Server/
enable=1
[VT]
name=VT
baseurl=http://192.168.0.1/repos/rhel5/VT/
enable=1
[Cluster]
name=Cluster
baseurl=http://192.168.0.1/repos/rhel5/Cluster/
enable=1
[ClusterStorage]
name=Cluster Storage
baseurl=http://192.168.0.1/repos/rhel5/ClusterStorage/
enable=1
";
#####################################################
files:
packages_ok::
# Create files/dirs and edit the new files
"/tftpboot/distro/RHEL/$(rh_distros)/."
create => "true";
"/tftpboot/distro/CentOS/$(centos_distros)/."
create => "true";
"$(dirs)/."
create => "true";
"/tftpboot/pxelinux.cfg/boot.msg"
create => "true",
perms => mo("644","root"),
edit_line => append_if_no_line("$(pxelinux_boot_menu)"),
edit_defaults => empty;
"/tftpboot/pxelinux.cfg/default"
create => "true",
perms => mo("644","root"),
edit_line => append_if_no_line("$(pxelinux_default)"),
edit_defaults => empty;
"/tftpboot/pxelinux.cfg/default.RHEL5.webserver"
create => "true",
perms => mo("644","root"),
edit_line => append_if_no_line("$(pxelinux_rhel5_webserver)"),
edit_defaults => empty;
"/tftpboot/kickstart/kickstart-RHEL5U2.cfg"
create => "true",
perms => mo("644","root"),
edit_line => append_if_no_line("$(kickstart_rhel5_conf)"),
edit_defaults => empty;
"/srv/www/repos/RHEL5.Base.repo"
create => "true",
perms => mo("644","root"),
edit_line => append_if_no_line("$(rhel5_base_repo)"),
edit_defaults => empty;
# Copy files
"/tftpboot"
copy_from => local_cp("/usr/share/syslinux"),
depth_search => recurse("inf"),
file_select => pxelinux_files,
action => immediate;
"$(tmp_location)"
perms => m("644"),
copy_from => local_cp("/var/cfengine/inputs"),
depth_search => recurse("inf"),
file_select => input_files,
action => immediate;
# Edit atftp, dhcp and apache2 configurations
"/etc/sysconfig/atftpd"
edit_line => append_if_no_line("$(atftpd_conf)"),
edit_defaults => empty,
classes => satisfied("atftpd_ready");
"/etc/sysconfig/dhcpd"
edit_line => append_if_no_line("$(dhcpd)"),
edit_defaults => empty;
"/etc/dhcpd.conf"
edit_line => append_if_no_line("$(dhcpd_conf)"),
edit_defaults => empty,
classes => satisfied("dhcpd_ready");
"/etc/apache2/httpd.conf"
edit_line => append_if_no_line("$(httpd_conf)"),
edit_defaults => std_defs,
classes => satisfied("apache2_ok");
# Make a static link
"/tftpboot/pxelinux.cfg/C0A8000C"
link_from => mylink("/tftpboot/pxelinux.cfg/default.RHEL5.webserver");
# Hash comment some lines for apaches
apache2_ok::
"/etc/apache2/httpd.conf"
edit_line => comment_lines_matching_apache2("#"),
classes => satisfied("apache2_ready");
commands:
# Restart services
atftpd_ready::
"/etc/init.d/atftpd restart";
dhcpd_ready::
"/etc/init.d/dhcpd restart";
apache2_ready::
"/etc/init.d/apache2 restart";
#####################################################
packages:
ipv4_192_168_0_1::
# Only the PXE boot server
"$(software)"
package_policy => "add",
package_method => zypper,
classes => satisfied("packages_ok");
}
body file_select pxelinux_files
{
leaf_name => { "pxelinux.0" };
file_result => "leaf_name";
}
body copy_from mycopy_local(from,server)
{
source => "$(from)";
compare => "digest";
}
body link_from mylink(x)
{
source => "$(x)";
link_type => "symlink";
}
body classes satisfied(new_class)
{
promise_kept => { "$(new_class)"};
promise_repaired => { "$(new_class)"};
}
bundle edit_line comment_lines_matching_apache2(comment)
{
vars:
"regex" slist => { "\s.*Options\sNone", "\s.*AllowOverride\sNone", "\s.*Deny\sfrom\sall" };
replace_patterns:
"^($(regex))$"
replace_with => comment("$(comment)");
}
body file_select input_files
{
leaf_name => { ".*.cf",".*.dat",".*.txt" };
file_result => "leaf_name";
}
Resolver management
bundle common g # globals
{
vars:
"searchlist" slist => {
"search iu.hio.no",
"search cfengine.com"
};
"nameservers" slist => {
"128.39.89.10",
"128.39.74.16",
"192.168.1.103"
};
classes:
"am_name_server" expression => reglist("@(nameservers)","$(sys.ipv4[eth1])");
}
body common control
{
any::
bundlesequence => {
"g",
resolver(@(g.searchlist),@(g.nameservers))
};
domain => "iu.hio.no";
}
bundle agent resolver(s,n)
{
files:
# When passing parameters down, we have to refer to
# a source context
"$(sys.resolv)" # test on "/tmp/resolv.conf" #
create => "true",
edit_line => doresolv("@(this.s)","@(this.n)"),
edit_defaults => reconstruct;
# or edit_defaults => modify
}
bundle edit_line doresolv(s,n)
{
vars:
"line" slist => { @(s), @(n) };
insert_lines:
"$(line)";
}
body edit_defaults reconstruct
{
empty_file_before_editing => "true";
edit_backup => "false";
max_file_size => "100000";
}
body edit_defaults modify
{
empty_file_before_editing => "false";
edit_backup => "false";
max_file_size => "100000";
}
Mount NFS filesystem
body common control
{
bundlesequence => { "mounts" };
}
bundle agent mounts
{
storage:
"/mnt" mount => nfs("slogans.iu.hio.no","/home");
}
body mount nfs(server,source)
{
mount_type => "nfs";
mount_source => "$(source)";
mount_server => "$(server)";
#mount_options => { "rw" };
edit_fstab => "true";
unmount => "true";
}
Unmount NFS filesystem
body common control
{
bundlesequence => { "mounts" };
}
bundle agent mounts
{
storage:
# Assumes the filesystem has been exported
"/mnt" mount => nfs("server.example.org","/home");
}
body mount nfs(server,source)
{
mount_type => "nfs";
mount_source => "$(source)";
mount_server => "$(server)";
edit_fstab => "true";
unmount => "true";
}
System security examples
- Distribute root passwords
- Distribute ssh keys
- Distribute ssh keys
Distribute root passwords
body common control
{
version => "1.2.3";
inputs => { "$(sys.libdir)/stdlib.cf" };
bundlesequence => { "SetRootPassword" };
}
bundle common g
{
vars:
"secret_keys_dir" string => "/tmp";
}
bundle agent SetRootPassword
{
vars:
# Or get variables directly from server with Enterprise
"remote-passwd" string => remotescalar("rem_password","127.0.0.1","yes");
# Test this on a copy
files:
"/var/cfengine/ppkeys/rootpw.txt"
copy_from => secure_cp("$(sys.fqhost)-root.txt","master_host.example.org");
# or $(pw_class)-root.txt
"/tmp/shadow"
edit_line => SetRootPw;
}
bundle edit_line SetRootPw
{
vars:
# Assume this file contains a single string of the form root:passwdhash:
# with : delimiters to avoid end of line/file problems
"pw" int => readstringarray("rpw","$(sys.workdir)/ppkeys/rootpw.txt",
"#[^\n]*",":","1","200");
field_edits:
"root:.*"
# Set field of the file to parameter
edit_field => col(":","2","$(rpw[root][1])","set");
}
bundle server passwords
{
vars:
# Read a file of format
#
# classname: host1,host2,host4,IP-address,regex.*,etc
#
"pw_classes" int => readstringarray("acl","$(g.secret_keys_dir)/classes.txt",
"#[^\n]*",":","100","4000");
"each_pw_class" slist => getindices("acl");
access:
"/secret/keys/$(each_pw_class)-root.txt"
admit => splitstring("$(acl[$(each_pw_class)][1])" , ":" , "100"),
ifencrypted => "true";
}
Distribute ssh keys
bundle agent allow_ssh_rootlogin_from_authorized_keys(user,sourcehost)
{
vars:
"local_cache" string => "/var/cfengine/ssh_cache";
"authorized_source" string => "/master/CFEngine/ssh_keys";
files:
"$(local_cache)/$(user).pub"
comment => "Copy public keys from a an authorized cache into a cache on localhost",
perms => mo("600","root"),
copy_from => remote_cp("$(authorized_source)/$(user).pub","$(sourcehost)"),
action => if_elapsed("60");
"/root/.ssh/authorized_keys"
comment => "Edit the authorized keys into the user's personal keyring",
edit_line => insert_file_if_no_line_matching("$(user)","$(local_cache)/$(user).pub"),
action => if_elapsed("60");
}
bundle agent allow_ssh_login_from_authorized_keys(user,sourcehost)
{
vars:
"local_cache" string => "/var/cfengine/ssh_cache";
"authorized_source" string => "/master/CFEngine/ssh_keys";
files:
"$(local_cache)/$(user).pub"
comment => "Copy public keys from a an authorized cache into a cache on localhost",
perms => mo("600","root"),
copy_from => remote_cp("$(authorized_source)/$(user).pub","$(sourcehost)"),
action => if_elapsed("60");
"/home/$(user)/.ssh/authorized_keys"
comment => "Edit the authorized keys into the user's personal keyring",
edit_line => insert_file_if_no_line_matching("$(user)","$(local_cache)/$(user).pub"),
action => if_elapsed("60");
}
bundle edit_line insert_file_if_no_line_matching(user,file)
{
classes:
"have_user" expression => regline("$(user).*","$(this.promiser)");
insert_lines:
!have_user::
"$(file)"
insert_type => "file";
}
System information examples
- Change detection
- Hashing for change detection (tripwire)
- Check filesystem space
- Class match example
- Global classes
- Logging
- Check filesystem space
Change detection
body common control
{
bundlesequence => { "testbundle" };
inputs => { "cfengine_stdlib.cf" };
}
bundle agent testbundle
{
files:
"/usr"
changes => detect_all_change,
depth_search => recurse("inf"),
action => background;
}
Hashing for change detection (tripwire)
Change detection is a powerful and easy way to monitor your environment, increase awareness and harden your system against security breaches.
body common control
{
bundlesequence => { "testbundle" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent testbundle
{
files:
"/home/mark/tmp/web" -> "me"
changes => detect_all_change,
depth_search => recurse("inf");
}
Check filesystem space
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
vars:
"free" int => diskfree("/tmp");
reports:
"Freedisk $(free)";
}
Class match example
body common control
{
bundlesequence => { "example" };
}
bundle agent example
{
classes:
"do_it" and => { classmatch(".*_3"), "linux" };
reports:
do_it::
"Host matches pattern";
}
Global classes
body common control
{
bundlesequence => { "g","tryclasses_1", "tryclasses_2" };
}
bundle common g
{
classes:
"one" expression => "any";
"client_network" expression => iprange("128.39.89.0/24");
}
bundle agent tryclasses_1
{
classes:
"two" expression => "any";
}
bundle agent tryclasses_2
{
classes:
"three" expression => "any";
reports:
one.three.!two::
"Success";
}
body common control
{
bundlesequence => { "g","tryclasses_1", "tryclasses_2" };
}
bundle common g
{
classes:
"one" expression => "any";
"client_network" expression => iprange("128.39.89.0/24");
}
bundle agent tryclasses_1
{
classes:
"two" expression => "any";
}
bundle agent tryclasses_2
{
classes:
"three" expression => "any";
reports:
one.three.!two::
"Success";
}
Logging
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
vars:
"software" slist => { "/root/xyz", "/tmp/xyz" };
files:
"$(software)"
create => "true",
action => logme("$(software)");
}
body action logme(x)
{
log_kept => "/tmp/private_keptlog.log";
log_failed => "/tmp/private_faillog.log";
log_repaired => "/tmp/private_replog.log";
log_string => "$(sys.date) $(x) promise status";
}
body common control
{
bundlesequence => { "one" };
}
bundle agent one
{
files:
"/tmp/xyz"
create => "true",
action => log;
}
body action log
{
log_level => "inform";
}
System administration examples
Centralized management
These examples show a simple setup for starting with a central approach to management of servers. Centralization of management is a simple approach suitable for small environments with few requirements. It is useful for clusters where systems are all alike.
All hosts the same
This shows the simplest approach in which all hosts are the same. It is too simple for most environments, but it serves as a starting point. Compare it to the next section that includes variation.
body common control
{
bundlesequence => { "central" };
}
bundle agent central
{
vars:
"policy_server" string => "myhost.domain.tld";
"mypackages" slist => {
"nagios"
"gcc",
"apache2",
"php5"
};
files:
# Password management can be very simple if all hosts are identical
"/etc/passwd"
comment => "Distribute a password file",
perms => mog("644","root","root"),
copy_from => secure_cp("/home/mark/LapTop/words/RoadAhead","$(policy_server)");
packages:
"$(mypackages)"
package_policy => "add",
package_method => generic;
# Add more promises below ...
}
body server control
{
allowconnects => { "127.0.0.1" , "::1", "10.20.30.0/24" };
allowallconnects => { "127.0.0.1" , "::1", "10.20.30.0/24" };
trustkeysfrom => { "127.0.0.1" , "::1", "10.20.30.0/24" };
# allowusers
}
bundle server my_access_rules()
{
access:
# myhost.domain.tld makes this file available to 10.20.30*
myhost_domain_tld::
"/etc/passwd"
admit => { "127.0.0.1", "10.20.30.0/24" };
}
Variation in hosts
body common control
{
bundlesequence => { "central" };
}
bundle agent central
{
classes:
"mygroup_1" or => { "myhost", "host1", "host2", "host3" };
"mygroup_2" or => { "host4", "host5", "host6" };
vars:
"policy_server" string => "myhost.domain.tld";
mygroup_1::
"mypackages" slist => {
"nagios"
"gcc",
"apache2",
"php5"
};
mygroup_2::
"mypackages" slist => {
"apache"
"mysql",
"php5"
};
files:
# Password management can be very simple if all hosts are identical
"/etc/passwd"
comment => "Distribute a password file",
perms => mog("644","root","root"),
copy_from => secure_cp("/etc/passwd","$(policy_server)");
packages:
"$(mypackages)"
package_policy => "add",
package_method => generic;
# Add more promises below ...
}
body server control
{
allowconnects => { "127.0.0.1" , "::1", "10.20.30.0/24" };
allowallconnects => { "127.0.0.1" , "::1", "10.20.30.0/24" };
trustkeysfrom => { "127.0.0.1" , "::1", "10.20.30.0/24" };
# allowusers
}
bundle server my_access_rules()
{
access:
# myhost.domain.tld makes this file available to 10.20.30*
myhost_domain_tld::
"/etc/passwd"
admit => { "127.0.0.1", "10.20.30.0/24" };
}
Updating from a central hub
The configuration bundled with the CFEngine source code contains an example of centralized updating of policy that covers more subtleties than this example, and handles fault tolerance. Here is the main idea behind it. For simplicity, we assume that all hosts are on network 10.20.30.* and that the central policy server/hub is 10.20.30.123.
bundle agent update
{
vars:
"master_location" string => "/var/cfengine/masterfiles";
"policy_server" string => "10.20.30.123",
comment => "IP address to locate your policy host.";
files:
"$(sys.workdir)/inputs"
perms => system("600"),
copy_from => remote_cp("$(master_location)",$(policy_server)),
depth_search => recurse("inf");
"$(sys.workdir)/bin"
perms => system("700"),
copy_from => remote_cp("/usr/local/sbin","localhost"),
depth_search => recurse("inf");
}
body server control
{
allowconnects => { "127.0.0.1" , "10.20.30.0/24" };
allowallconnects => { "127.0.0.1" , "10.20.30.0/24" };
trustkeysfrom => { "127.0.0.1" , "10.20.30.0/24" };
}
bundle server my_access_rules()
{
access:
10_20_30_123::
"/var/cfengine/masterfiles"
admit => { "127.0.0.1", "10.20.30.0/24" };
}
Laptop support configuration
Laptops do not need a lot of confguration support. IP addresses are set by DHCP and conditions are changeable. But you want to set your DNS search domains to familiar settings in spite of local DHCP configuration, and another useful trick is to keep a regular backup of disk changes on the local disk. This won't help against disk destruction, but it is a huge advantage when your user accidentally deletes files while travelling or offline.
body common control
{
bundlesequence => {
"update",
"garbage_collection",
"main",
"backup",
};
inputs => {
"update.cf",
"site.cf",
"library.cf"
};
}
body agent control
{
# if default runtime is 5 mins we need this for long jobs
ifelapsed => "15";
}
body monitor control
{
forgetrate => "0.7";
}
body executor control
{
splaytime => "1";
mailto => "mark@iu.hio.no";
smtpserver => "localhost";
mailmaxlines => "30";
# Instead of a separate update script, now do this
exec_command => "$(sys.workdir)/bin/cf-agent -f failsafe.cf && $(sys.workdir)/bin/cf-agent";
}
bundle agent main
{
vars:
"component" slist => { "cf-monitord", "cf-serverd" };
# - - - - - - - - - - - - - - - - - - - - - - - -
files:
"$(sys.resolv)" # test on "/tmp/resolv.conf" #
create => "true",
edit_line => resolver,
edit_defaults => def;
processes:
"$(component)" restart_class => canonify("$(component)_not_running");
# - - - - - - - - - - - - - - - - - - - - - - - -
commands:
"$(sys.workdir)/bin/$(component)"
if => canonify("$(component)_not_running");
}
bundle agent backup
{
files:
"/home/backup"
copy_from => cp("/home/mark"),
depth_search => recurse("inf"),
file_select => exclude_files,
action => longjob;
}
bundle agent garbage_collection
{
files:
"$(sys.workdir)/outputs"
delete => tidy,
file_select => days_old("3"),
depth_search => recurse("inf");
}
Process management
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
processes:
"sleep"
signals => { "term", "kill" };
}
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
processes:
"sleep"
process_count => up("sleep");
reports:
sleep_out_of_control::
"Out of control";
}
body process_count up(s)
{
match_range => "5,10"; # or irange("1","10");
out_of_range_define => { "$(s)_out_of_control" };
}
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
processes:
".*"
process_select => proc_finder("a.*"),
process_count => up("cfservd");
}
body process_count up(s)
{
match_range => "1,10"; # or irange("1","10");
out_of_range_define => { "$(s)_out_of_control" };
}
body process_select proc_finder(p)
{
stime_range => irange(ago("0","0","0","2","0","0"),now);
process_result => "stime";
}
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
processes:
".*"
process_select => proc_finder("a.*"),
process_count => up("cfservd");
}
body process_count up(s)
{
match_range => "1,10"; # or irange("1","10");
out_of_range_define => { "$(s)_out_of_control" };
}
body process_select proc_finder(p)
{
process_owner => { "avahi", "bin" };
command => "$(p)";
pid => "100,199";
vsize => "0,1000";
process_result => "command.(process_owner|vsize)";
}
body common control
{
bundlesequence => { "process_restart" };
}
bundle agent process_restart
{
processes:
"/usr/bin/daemon"
restart_class => "launch";
commands:
launch::
"/usr/bin/daemon";
}
body common control
{
bundlesequence => { "process_restart" };
}
bundle agent process_restart
{
vars:
"component"
slist => {
"cf-monitord",
"cf-serverd",
"cf-execd"
};
processes:
"$(component)"
restart_class => canonify("$(component)_not_running");
commands:
"/var/cfengine/bin/$(component)"
if => canonify("$(component)_not_running");
}
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
processes:
"cfservd"
process_count => up("cfservd");
cfservd_out_of_control::
"cfservd"
signals => { "stop" , "term" },
restart_class => "stopped_out_of_control_cf_serverd";
commands:
stopped_out_of_control_cf_serverd::
"/usr/local/sbin/cfservd";
}
body process_count up(s)
{
match_range => "1,10"; # or irange("1","10");
out_of_range_define => { "$(s)_out_of_control" };
}
Kill process
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
processes:
"sleep"
signals => { "term", "kill" };
}
Restart process
A basic pattern for restarting processes:
body common control
{
bundlesequence => { "process_restart" };
}
bundle agent process_restart
{
processes:
"/usr/bin/daemon"
restart_class => "launch";
commands:
launch::
"/usr/bin/daemon";
}
This can be made more sophisticated to handle generic lists:
body common control
{
bundlesequence => { "process_restart" };
}
bundle agent process_restart
{
vars:
"component"
slist => {
"cf-monitord",
"cf-serverd",
"cf-execd"
};
processes:
"$(component)"
restart_class => canonify("not_running_$(component)");
commands:
"/var/cfengine/bin/$(component)"
if => canonify("not_running_$(component)");
}
Why? Separating this into two parts gives a high level of control and conistency to CFEngine. There are many options for command execution, like the ability to run commands in a sandbox or as setuid
. These should not be reproduced in processes.
Mount a filesystem
body common control
{
bundlesequence => { "mounts" };
}
bundle agent mounts
{
storage:
"/mnt" mount => nfs("slogans.iu.hio.no","/home");
}
body mount nfs(server,source)
{
mount_type => "nfs";
mount_source => "$(source)";
mount_server => "$(server)";
#mount_options => { "rw" };
edit_fstab => "true";
unmount => "true";
}
Manage a system process
Ensure running
Ensure not running
Prune processes
Ensure running
The simplest example might look like this:
bundle agent restart_process
{
processes:
"httpd"
comment => "Make sure apache web server is running",
restart_class => "restart_httpd";
commands:
restart_httpd::
"/etc/init.d/apache2 restart";
}
This example shows how the CFEngine components could be started using a pattern.
bundle agent CFEngine_processes
{
vars:
"component" slist => { "cf-execd", "cf-monitord", "cf-serverd", "cf-hub" };
processes:
"$(component)"
comment => "Make sure server parts of CFEngine are running",
restart_class => canonify("$(component)_not_running");
commands:
"$(sys.workdir)/bin/$(component)"
comment => "Make sure server parts of CFEngine are running",
if => canonify("$(component)_not_running");
}
Ensure not running
bundle agent restart_process
{
vars:
"killprocs" slist => { "snmpd", "gameserverd", "irc", "crack" };
processes:
"$(killprocs)"
comment => "Ensure processes are not running",
signals => { "term", "kill" };
}
Prune processes
This example kills processes owned by a particular user that have exceeded 100000 bytes of resident memory.
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
processes:
".*"
process_select => big_processes("mark"),
signals => { "term" };
}
body process_select big_processes(o)
{
process_owner => { $(o) };
rsize => irange("100000","900000");
process_result => "rsize.process_owner";
}
Set up HPC clusters
HPC cluster machines are usually all identical, so the CFEngine configuration is very simple. HPC clients value CPU and memory resources, so we can shut down unnecessary services to save CPU. We can also change the scheduling rate of CFEngine to run less frequently, and save a little:
body executor control
{
splaytime => "1";
mailto => "cfengine@example.com";
smtpserver => "localhost";
mailmaxlines => "30";
# Once per hour, on the hour
schedule => { "Min00" };
}
bundle agent services_disable
{
vars:
# list all of xinetd services (case sensitive)
"xinetd_services" slist => {
"imap",
"imaps",
"ipop2",
"ipop3",
"krb5-telnet",
"klogin",
"kshell",
"ktalk",
"ntalk",
"pop3s",
};
methods:
# perform the actual disable all xinetd services according to the list above
"any" usebundle => disable_xinetd("$(xinetd_services)");
processes:
"$(xinetd_services)"
signals => { "kill" };
}
bundle agent disable_xinetd(name)
{
vars:
"status" string => execresult("/sbin/chkconfig --list $(name)", "useshell");
classes:
"on" expression => regcmp(".*on.*","$(status)");
commands:
on::
"/sbin/chkconfig $(name) off",
comment => "disable $(name) service";
reports:
on::
"disable $(name) service.";
}
Set up name resolution
There are many ways to do name resolution setup1 We write a reusable bundle using the editing features.
A simple and straightforward approach is to maintain a separate modular bundle for this task. This avoids too many levels of abstraction and keeps all the information in one place. We implement this as a simple editing promise for the /etc/resolv.conf file.
bundle agent system_files
{
files:
"$(sys.resolv)" # test on "/tmp/resolv.conf" #
comment => "Add lines to the resolver configuration",
create => "true",
edit_line => resolver,
edit_defaults => std_edits;
# ...other system files ...
}
bundle edit_line resolver
{
delete_lines:
# delete any old name servers or junk we no longer need
"search.*";
"nameserver 80.65.58.31";
"nameserver 80.65.58.32";
"nameserver 82.103.128.146";
"nameserver 78.24.145.4";
"nameserver 78.24.145.5";
"nameserver 128.39.89.10";
insert_lines:
"search mydomain.tld" location => start;
special_net::
"nameserver 128.39.89.8";
"nameserver 128.39.74.66";
!special_net::
"nameserver 128.38.34.12";
any::
"nameserver 212.112.166.18";
"nameserver 212.112.166.22";
}
A second approach is to try to conceal the operational details behind a veil of abstraction.
bundle agent system_files
{
vars:
"searchlist" string => "iu.hio.no CFEngine.com";
"nameservers" slist => {
"128.39.89.10",
"128.39.74.16",
"192.168.1.103"
};
files:
"$(sys.resolv)" # test on "/tmp/resolv.conf" #
create => "true",
edit_line => doresolv("$(s)","@(this.n)"),
edit_defaults => empty;
# ....
}
bundle edit_line doresolv(search,names)
{
insert_lines:
"search $(search)";
"nameserver $(names)";
}
bundle agent system_files { # ...
files: "/etc/hosts" comment => "Add hosts to the /etc/hosts file", edit_line => fix_etc_hosts; }
bundle edit_line fix_etc_hosts { vars: "names[127.0.0.1]" string => "localhost localhost.CFEngine.com"; "names[128.39.89.12]" string => "myhost myhost.CFEngine.com"; "names[128.39.89.13]" string => "otherhost otherhost.CFEngine.com"; # etc
"i" slist => getindices("names");
insert_lines: "$(i) $(names[$(i)])"; } ```
DNS is not the only name service, of course. Unix has its older /etc/hosts file which can also be managed using file editing. We simply append this to the system_files bundle.
bundle agent system_files
{
vars:
"searchlist" string => "iu.hio.no CFEngine.com";
"nameservers" slist => {
"128.39.89.10",
"128.39.74.16",
"192.168.1.103"
};
files:
"$(sys.resolv)" # test on "/tmp/resolv.conf" #
create => "true",
edit_line => doresolv("$(s)","@(this.n)"),
edit_defaults => empty;
# ....
}
bundle edit_line doresolv(search,names)
{
insert_lines:
"search $(search)";
"nameserver $(names)";
}
bundle agent system_files { # ...
files: "/etc/hosts" comment => "Add hosts to the /etc/hosts file", edit_line => fix_etc_hosts; }
bundle edit_line fix_etc_hosts { vars: "names[127.0.0.1]" string => "localhost localhost.CFEngine.com"; "names[128.39.89.12]" string => "myhost myhost.CFEngine.com"; "names[128.39.89.13]" string => "otherhost otherhost.CFEngine.com"; # etc
"i" slist => getindices("names");
insert_lines: "$(i) $(names[$(i)])"; } ```
Set up sudo
Setting up sudo is straightforward, and is best managed by copying trusted files from a repository.
bundle agent system_files
{
vars:
"masterfiles" string => "/subversion_projects/masterfiles";
# ...
files:
"/etc/sudoers"
comment => "Make sure the sudo configuration is secure and up to date",
perms => mog("440","root","root"),
copy_from => secure_cp("$(masterfiles)/sudoers","$(policy_server)");
}
Environments (virtual)
body common control
{
bundlesequence => { "my_vm_cloud" };
}
bundle agent my_vm_cloud
{
vars:
"vms[atlas]" slist => { "guest1", "guest2", "guest3" };
environments:
scope||any:: # These should probably be in class "any" to ensure uniqueness
"$(vms[$(sys.host)])"
environment_resources => virt_xml("$(xmlfile[$(this.promiser)])"),
environment_interface => vnet("eth0,192.168.1.100/24"),
environment_type => "test",
environment_host => "atlas";
# default environment_state => "create" on host, and "suspended elsewhere"
}
body environment_resources virt_xml(specfile)
{
env_spec_file => "$(specfile)";
}
body environment_interface vnet(primary)
{
env_name => "$(this.promiser)";
env_addresses => { "$(primary)" };
host1::
env_network => "default_vnet1";
host2::
env_network => "default_vnet2";
}
Environment variables
body common control
{
bundlesequence => { "my_vm_cloud" };
}
bundle agent my_vm_cloud
{
environments:
"centos5"
environment_resources => virt_xml,
environment_type => "xen",
environment_host => "ursa-minor";
# default environment_state => "create" on host, and "suspended elsewhere"
}
body environment_resources virt_xml
{
env_spec_file => "/srv/xen/centos5-libvirt-create.xml";
}
Tidying garbage files
Emulating the tidy
feature of CFEngine 2.
body common control
{
any::
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/tmp/test"
delete => tidy,
file_select => zero_age,
depth_search => recurse("inf");
}
body depth_search recurse(d)
{
#include_basedir => "true";
depth => "$(d)";
}
body delete tidy
{
dirlinks => "delete";
rmdirs => "false";
}
body file_select zero_age
{
mtime => irange(ago(1,0,0,0,0,0),now);
file_result => "mtime";
}
System file examples
Editing password or group files
To change the password of a system, we need to edit a file. A file is a complex object - once open there is a new world of possible promises to make about its contents. CFEngine has bundles of promises that are specially for editing.
body common control
{
inputs => { "$(sys.libdir)/stdlib.cf" };
bundlesequence => { "edit_passwd" };
}
bundle agent edit_passwd
{
vars:
"userset" slist => { "user1", "user2", "user3" };
files:
"/etc/passwd"
edit_line => set_user_field("mark","7","/set/this/shell");
"/etc/group"
edit_line => append_user_field("root","4","@(main.userset)");
}
Editing password or group files custom
In this example the bundles from the Community Open Promise-Body Library are included directly in the policy instead of being input as a separate file.
body common control
{
bundlesequence => { "addpasswd" };
}
bundle agent addpasswd
{
vars:
# want to set these values by the names of their array keys
"pwd[mark]" string => "mark:x:1000:100:Mark Burgess:/home/mark:/bin/bash";
"pwd[fred]" string => "fred:x:1001:100:Right Said:/home/fred:/bin/bash";
"pwd[jane]" string => "jane:x:1002:100:Jane Doe:/home/jane:/bin/bash";
files:
"/tmp/passwd"
create => "true",
edit_line => append_users_starting("addpasswd.pwd");
}
bundle edit_line append_users_starting(v)
{
vars:
"index" slist => getindices("$(v)");
classes:
"add_$(index)" not => userexists("$(index)");
insert_lines:
"$($(v)[$(index)])",
if => "add_$(index)";
}
bundle edit_line append_groups_starting(v)
{
vars:
"index" slist => getindices("$(v)");
classes:
"add_$(index)" not => groupexists("$(index)");
insert_lines:
"$($(v)[$(index)])",
if => "add_$(index)";
}
Log rotation
body common control
{
bundlesequence => { "testbundle" };
}
bundle agent testbundle
{
files:
"/home/mark/tmp/rotateme"
rename => rotate("4");
}
body rename rotate(level)
{
rotate => "$(level)";
}
Garbage collection
body common control
{
bundlesequence => { "garbage_collection" };
inputs => { "cfengine_stdlib.cf" };
}
bundle agent garbage_collection
{
files:
Sunday::
"$(sys.workdir)/nova_repair.log"
comment => "Rotate the promises repaired logs each week",
rename => rotate("7"),
action => if_elapsed("10000");
"$(sys.workdir)/nova_notkept.log"
comment => "Rotate the promises not kept logs each week",
rename => rotate("7"),
action => if_elapsed("10000");
"$(sys.workdir)/promise.log"
comment => "Rotate the promises not kept logs each week",
rename => rotate("7"),
action => if_elapsed("10000");
any::
"$(sys.workdir)/outputs"
comment => "Garbage collection of any output files",
delete => tidy,
file_select => days_old("3"),
depth_search => recurse("inf");
"$(sys.workdir)/"
comment => "Garbage collection of any output files",
delete => tidy,
file_select => days_old("14"),
depth_search => recurse("inf");
# Other resources
"/tmp"
comment => "Garbage collection of any temporary files",
delete => tidy,
file_select => days_old("3"),
depth_search => recurse("inf");
"/var/log/apache2/.*bz"
comment => "Garbage collection of rotated log files",
delete => tidy,
file_select => days_old("30"),
depth_search => recurse("inf");
"/var/log/apache2/.*gz"
comment => "Garbage collection of rotated log files",
delete => tidy,
file_select => days_old("30"),
depth_search => recurse("inf");
"/var/log/zypper.log"
comment => "Prevent the zypper log from choking the disk",
rename => rotate("0"),
action => if_elapsed("10000");
}
Manage a system file
Simple template
Simple versioned template
Macro template
Custom editing
Simple template
bundle agent hand_edited_config_file
{
vars:
"file_template" string =>
"
127.0.0.1 localhost
::1 localhost ipv6-localhost ipv6-loopback
fe00::0 ipv6-localnet
ff00::0 ipv6-mcastprefix
ff02::1 ipv6-allnodes
ff02::2 ipv6-allrouters
ff02::3 ipv6-allhosts
10.0.0.100 host1.domain.tld host1
10.0.0.101 host2.domain.tld host2
10.0.0.20 host3.domain.tld host3
10.0.0.21 host4.domain.tld host4
";
##############################################################
files:
"/etc/hosts"
comment => "Define the content of all host files from this master source",
create => "true",
edit_line => append_if_no_lines("$(file_template)"),
edit_defaults => empty,
perms => mo("$(mode)","root"),
action => if_elapsed("60");
}
Simple versioned template
The simplest approach to managing a file is to maintain a master copy by hand, keeping it in a version controlled repository (e.g. svn), and installing this version on the end machine.
We'll assume that you have a version control repository that is located on some independent server, and has been checked out manually once (with authentication) in /mysite/masterfiles.
bundle agent hand_edited_config_file
{
vars:
"masterfiles" string => "/mysite/masterfiles";
"policy_server" string => "policy_host.domain.tld";
files:
"/etc/hosts"
comment => "Synchronize hosts with a hand-edited template in svn",
perms => m("644"),
copy_from => remote_cp("$(masterfiles)/trunk/hosts_master","$(policy_server)");
commands:
"/usr/bin/svn update"
comment => "Update the company document repository including manuals to a local copy",
contain => silent_in_dir("$(masterfiles)/trunk"),
if => canonify("$(policy_server)");
}
Macro template
The next simplest approach to file management is to add variables to the template that will be expanded into local values at the end system, e.g. using variables like $(sys.host)
for the name of the host within the body of the versioned template.
bundle agent hand_edited_template
{
vars:
"masterfiles" string => "/mysite/masterfiles";
"policy_server" string => "policy_host.domain.tld";
files:
"/etc/hosts"
comment => "Synchronize hosts with a hand-edited template in svn",
perms => m("644"),
create => "true",
edit_line => expand_template("$(masterfiles)/trunk/hosts_master"),
edit_defaults => empty,
action => if_elapsed("60");
commands:
"/usr/bin/svn update"
comment => "Update the company document repository including manuals to a local copy",
contain => silent_in_dir("$(masterfiles)/trunk"),
if => canonify("$(policy_server)");
}
The macro template file may contain variables, as below, that get expanded by CFEngine.
bundle agent hand_edited_template
{
vars:
"masterfiles" string => "/mysite/masterfiles";
"policy_server" string => "policy_host.domain.tld";
files:
"/etc/hosts"
comment => "Synchronize hosts with a hand-edited template in svn",
perms => m("644"),
create => "true",
edit_line => expand_template("$(masterfiles)/trunk/hosts_master"),
edit_defaults => empty,
action => if_elapsed("60");
commands:
"/usr/bin/svn update"
comment => "Update the company document repository including manuals to a local copy",
contain => silent_in_dir("$(masterfiles)/trunk"),
if => canonify("$(policy_server)");
}
127.0.0.1 localhost $(sys.host) ::1 localhost ipv6-localhost ipv6-loopback fe00::0 ipv6-localnet ff00::0 ipv6-mcastprefix ff02::1 ipv6-allnodes ff02::2 ipv6-allrouters ff02::3 ipv6-allhosts 10.0.0.100 host1.domain.tld host1 10.0.0.101 host2.domain.tld host2 10.0.0.20 host3.domain.tld host3 10.0.0.21 host4.domain.tld host4
$(definitions.more_hosts) ```
Custom editing
If you do not control the starting state of the file, because it is distributed by an operating system vendor for instance, then editing the final state is the best approach. That way, you will get changes that are made by the vendor, and will ensure your own modifications are kept even when updates arrive.
bundle agent modifying_managed_file
{
vars:
"data" slist => { "10.1.2.3 sirius", "10.1.2.4 ursa-minor", "10.1.2.5 orion"};
files:
"/etc/hosts"
comment => "Append a list of lines to the end of a file if they don't exist",
perms => m("644"),
create => "true",
edit_line => append_if_no_lines("modifying_managed_file.data"),
action => if_elapsed("60");
}
Another example shows how to set the values of variables using a data-driven approach and methods from the standard library.
body common control
{
bundlesequence => { "testsetvar" };
}
bundle agent testsetvar
{
vars:
"v[variable_1]" string => "value_1";
"v[variable_2]" string => "value_2";
files:
"/tmp/test_setvar"
edit_line => set_variable_values("testsetvar.v");
}
Windows registry examples
Windows registry
body common control
{
bundlesequence => { "reg" };
}
bundle agent reg
{
vars:
"value" string => registryvalue("HKEY_LOCAL_MACHINE\SOFTWARE\Cfengine AS\Cfengine","value3");
reports:
windows::
"Value extracted: $(value)";
}
unit_registry_cache.cf
body common control
{
bundlesequence => {
# "registry_cache"
# "registry_restore"
};
}
bundle agent registry_cache
{
databases:
windows::
"HKEY_LOCAL_MACHINE\SOFTWARE\Adobe"
database_operation => "cache",
database_type => "ms_registry",
comment => "Save correct registry settings for Adobe products";
}
bundle agent registry_restore
{
databases:
windows::
"HKEY_LOCAL_MACHINE\SOFTWARE\Adobe"
database_operation => "restore",
database_type => "ms_registry",
comment => "Make sure Adobe products have correct registry settings";
}
unit_registry.cf
body common control
{
bundlesequence => { "databases" };
}
bundle agent databases
{
databases:
windows::
# Registry has (value,data) pairs in "keys" which are directories
# "HKEY_LOCAL_MACHINE\SOFTWARE\Cfengine AS"
# database_operation => "create",
# database_type => "ms_registry";
# "HKEY_LOCAL_MACHINE\SOFTWARE\Cfengine AS\Cfengine"
# database_operation => "create",
# database_rows => { "value1,REG_SZ,new value 1", "value2,REG_SZ,new val 2"} ,
# database_type => "ms_registry";
"HKEY_LOCAL_MACHINE\SOFTWARE\Cfengine AS\Cfengine"
database_operation => "delete",
database_columns => { "value1", "value2" } ,
database_type => "ms_registry";
# "HKEY_LOCAL_MACHINE\SOFTWARE\Cfengine AS\Cfengine"
# database_operation => "cache", # cache,restore
# registry_exclude => { ".*Windows.*CurrentVersion.*", ".*Touchpad.*", ".*Capabilities.FileAssociations.*", ".*Rfc1766.*" , ".*Synaptics.SynTP.*", ".*SupportedDevices.*8086", ".*Microsoft.*ErrorThresholds" },
# database_type => "ms_registry";
"HKEY_LOCAL_MACHINE\SOFTWARE\Cfengine AS"
database_operation => "restore",
database_type => "ms_registry";
}
File permissions
ACL file example
body common control
{
bundlesequence => { "acls" };
}
bundle agent acls
{
files:
"/media/flash/acl/test_dir"
depth_search => include_base,
acl => template;
}
body acl template
{
acl_method => "overwrite";
acl_type => "posix";
acl_directory_inherit => "parent";
aces => { "user:*:r(wwx),-r:allow", "group:*:+rw:allow", "mask:x:allow", "all:r"};
}
body acl win
{
acl_method => "overwrite";
acl_type => "ntfs";
acl_directory_inherit => "nochange";
aces => { "user:Administrator:rw", "group:Bad:rwx(Dpo):deny" };
}
body depth_search include_base
{
include_basedir => "true";
}
ACL generic example
body common control
{
bundlesequence => { "acls" };
}
bundle agent acls
{
files:
"/media/flash/acl/test_dir"
depth_search => include_base,
acl => test;
}
body acl test
{
acl_type => "generic";
aces => {"user:bob:rwx", "group:staff:rx", "all:r"};
}
body depth_search include_base
{
include_basedir => "true";
}
ACL secret example
body common control
{
bundlesequence => { "acls" };
}
bundle agent acls
{
files:
windows::
"c:\Secret"
acl => win,
depth_search => include_base,
comment => "Secure the secret directory from unauthorized access";
}
body acl win
{
acl_method => "overwrite";
aces => { "user:Administrator:rwx" };
}
body depth_search include_base
{
include_basedir => "true";
}
User management examples
Local user management
There are many approaches to managing users. You can edit system files
like /etc/passwd
directly, you can use commands on some systems like
useradd
. However the easiest, and preferred way is to use
CFEngine's native users
type promise.
Ensuring a local user has a specific password
This example shows ensuring that the local users root
is managed if
there is a specific password hash defined.
body file control
{
# This policy uses parts of the standard library.
inputs => { "$(sys.libdir)/users.cf" };
}
bundle agent main
{
vars:
# This is the hashed password for 'vagrant'
debian_8::
"root_hash"
string => "$6$1nRTeNoE$DpBSe.eDsuZaME0EydXBEf.DAwuzpSoIJhkhiIAPgRqVKlmI55EONfvjZorkxNQvK2VFfMm9txx93r2bma/4h/";
users:
linux::
"root"
policy => "present",
password => hashed_password( $(root_hash) ),
if => isvariable("root_hash");
}
This policy can be found in
/var/cfengine/share/doc/examples/local_user_password.cf
and downloaded directly from
github.
root@debian-jessie:/core/examples# grep root /etc/shadow
root:!:16791:0:99999:7:::
root@debian-jessie:/core/examples# cf-agent -KIf ./local_user_password.cf
info: User promise repaired
root@debian-jessie:/core/examples# grep root /etc/shadow
root:$6$1nRTeNoE$DpBSe.eDsuZaME0EydXBEf.DAwuzpSoIJhkhiIAPgRqVKlmI55EONfvjZorkxNQvK2VFfMm9txx93r2bma/4h/:16791:0:99999:7:::
Ensuring local users are present
This example shows ensuring that the local users jack
and jill
are
present on all linux systems using the native users
type promise.
body file control
{
# This policy uses parts of the standard library.
inputs => { "$(sys.libdir)/files.cf" };
}
bundle agent main
{
vars:
"users" slist => { "jack", "jill" };
"skel" string => "/etc/skel";
users:
linux::
"$(users)"
home_dir => "/home/$(users)",
policy => "present",
home_bundle => home_skel( $(users), $(skel) );
}
bundle agent home_skel(user, skel)
{
files:
"/home/$(user)/."
create => "true",
copy_from => seed_cp( $(skel) ),
depth_search => recurse( "inf" );
}
This policy can be found in
/var/cfengine/share/doc/examples/local_users_present.cf
and downloaded directly from
github.
Lets check the environment to see that the users do not currently exist.
root@debian-jessie:/CFEngine/core/examples# egrep "jack|jill" /etc/passwd
root@debian-jessie:/core/examples# ls -al /home/{jack,jill}
ls: cannot access /home/jack: No such file or directory
ls: cannot access /home/jill: No such file or directory
Let's run the policy and inspect the state of the system afterwards.
root@debian-jessie:/core/examples# cf-agent -KIf ./users_present.cf
info: Created directory '/home/jack/.'
info: Copying from 'localhost:/etc/skel/.bashrc'
info: Copying from 'localhost:/etc/skel/.profile'
info: Copying from 'localhost:/etc/skel/.bash_logout'
info: User promise repaired
info: Created directory '/home/jill/.'
info: Copying from 'localhost:/etc/skel/.bashrc'
info: Copying from 'localhost:/etc/skel/.profile'
info: Copying from 'localhost:/etc/skel/.bash_logout'
info: User promise repaired
root@debian-jessie:/core/examples# egrep "jack|jill" /etc/passwd
jack:x:1001:1001::/home/jack:/bin/sh
jill:x:1002:1002::/home/jill:/bin/sh
root@debian-jessie:/core/examples# ls -al /home/{jack,jill}
/home/jack:
total 20
drwxr-xr-x 2 root root 4096 Dec 22 16:37 .
drwxr-xr-x 5 root root 4096 Dec 22 16:37 ..
-rw-r--r-- 1 root root 220 Dec 22 16:37 .bash_logout
-rw-r--r-- 1 root root 3515 Dec 22 16:37 .bashrc
-rw-r--r-- 1 root root 675 Dec 22 16:37 .profile
/home/jill:
total 20
drwxr-xr-x 2 root root 4096 Dec 22 16:37 .
drwxr-xr-x 5 root root 4096 Dec 22 16:37 ..
-rw-r--r-- 1 root root 220 Dec 22 16:37 .bash_logout
-rw-r--r-- 1 root root 3515 Dec 22 16:37 .bashrc
-rw-r--r-- 1 root root 675 Dec 22 16:37 .profile
Ensuring local users are locked
This example shows ensuring that the local users jack
and jill
are
locked if they are present on linux systems using the native users
type promise.
bundle agent main
{
vars:
"users" slist => { "jack", "jill" };
users:
linux::
"$(users)"
policy => "locked";
}
This policy can be found in
/var/cfengine/share/doc/examples/local_users_locked.cf
and downloaded directly from
github.
This output shows the state of the /etc/shadow
file before running
the example policy:
root@debian-jessie:/core/examples# egrep "jack|jill" /etc/shadow
jack:x:16791:0:99999:7:::
jill:x:16791:0:99999:7:::
root@debian-jessie:/core/examples# cf-agent -KIf ./local_users_locked.cf
info: User promise repaired
info: User promise repaired
root@debian-jessie:/core/examples# egrep "jack|jill" /etc/shadow
jack:!x:16791:0:99999:7::1:
jill:!x:16791:0:99999:7::1:
Ensuring local users are absent
This example shows ensuring that the local users jack
and jill
are
absent on linux systems using the native users
type promise.
bundle agent main
{
vars:
"users" slist => { "jack", "jill" };
users:
linux::
"$(users)"
policy => "absent";
}
This policy can be found in
/var/cfengine/share/doc/examples/local_users_absent.cf
and downloaded directly from
github.
Before activating the example policy, lets inspect the current state of the system.
root@debian-jessie:/core/examples# egrep "jack|jill" /etc/passwd
jack:x:1001:1001::/home/jack:/bin/sh
jill:x:1002:1002::/home/jill:/bin/sh
root@debian-jessie:/core/examples# ls -al /home/{jack,jill}
/home/jack:
total 20
drwxr-xr-x 2 root root 4096 Dec 22 16:37 .
drwxr-xr-x 5 root root 4096 Dec 22 16:37 ..
-rw-r--r-- 1 root root 220 Dec 22 16:37 .bash_logout
-rw-r--r-- 1 root root 3515 Dec 22 16:37 .bashrc
-rw-r--r-- 1 root root 675 Dec 22 16:37 .profile
/home/jill:
total 20
drwxr-xr-x 2 root root 4096 Dec 22 16:37 .
drwxr-xr-x 5 root root 4096 Dec 22 16:37 ..
-rw-r--r-- 1 root root 220 Dec 22 16:37 .bash_logout
-rw-r--r-- 1 root root 3515 Dec 22 16:37 .bashrc
-rw-r--r-- 1 root root 675 Dec 22 16:37 .profile
From the above output we can see that the local users jack
and
jill
are present, and that they both have home directories.
Now lets activate the example policy and insepect the result.
root@debian-jessie:/core/examples# cf-agent -KIf ./local_users_absent.cf
info: User promise repaired
info: User promise repaired
root@debian-jessie:/core/examples# egrep "jack|jill" /etc/passwd
root@debian-jessie:/core/examples# ls -al /home/{jack,jill}
/home/jack:
total 20
drwxr-xr-x 2 root root 4096 Dec 22 16:37 .
drwxr-xr-x 5 root root 4096 Dec 22 16:37 ..
-rw-r--r-- 1 root root 220 Dec 22 16:37 .bash_logout
-rw-r--r-- 1 root root 3515 Dec 22 16:37 .bashrc
-rw-r--r-- 1 root root 675 Dec 22 16:37 .profile
/home/jill:
total 20
drwxr-xr-x 2 root root 4096 Dec 22 16:37 .
drwxr-xr-x 5 root root 4096 Dec 22 16:37 ..
-rw-r--r-- 1 root root 220 Dec 22 16:37 .bash_logout
-rw-r--r-- 1 root root 3515 Dec 22 16:37 .bashrc
-rw-r--r-- 1 root root 675 Dec 22 16:37 .profile
From the above output we can see that the local users jack
and
jill
were removed from the system as desired. Note that their home
directories remain, and if we wanted them to be purged we would have
to have a separate promise to perform that cleanup.
Local group management
CFEngine does not currently have a native groups
type promise so you
will need to either edit the necessary files using files
type
promises, or arrange for the proper commands to be run in order to
create or delete groups.
Ensure a local group is present
Add lines to the password file, and users to group if they are not already there.
This example uses the native operating system commands to show ensuring that a group is present.
body file control
{
# This policy uses parts of the standard library.
inputs => { "$(sys.libdir)/paths.cf" };
}
bundle agent main
{
classes:
"group_cfengineers_absent"
not => groupexists("cfengineers");
commands:
linux.group_cfengineers_absent::
"$(paths.groupadd)"
args => "cfengineers";
}
This policy can be found in
/var/cfengine/share/doc/examples/local_group_present.cf
and downloaded directly from
github.
First lets inspect the current state of the system.
root@debian-jessie:/core/examples# grep cfengineers /etc/group
Now lets activate the example policy and check the resulting state of the system.
root@debian-jessie:/core/examples# cf-agent -KIf ./local_group_present.cf
info: Executing 'no timeout' ... '/usr/sbin/groupadd cfengineers'
info: Completed execution of '/usr/sbin/groupadd cfengineers'
root@debian-jessie:/CFEngine/core2.git/examples# grep cfengineers /etc/group
cfengineers:x:1001:
Ensureing a user is a member of a secondary group
This example shows using the native users
type promise to ensure
that a user is a member of a particular group.
bundle agent main
{
users:
linux::
"jill"
policy => "present",
groups_secondary => { "cfengineers" };
}
This policy can be found in
/var/cfengine/share/doc/examples/local_user_secondary_group_member.cf
and downloaded directly from
github.
First lets inspect the current state of the system
root@debian-jessie:/core/examples# grep jill /etc/passwd
root@debian-jessie:/core/examples# grep jill /etc/group
Now lets actiavte the example policy and inspect the resulting state.
root@debian-jessie:/core/examples# cf-agent -KIf ./local_user_secondary_group_member.cf
info: User promise repaired
root@debian-jessie:/core/examples# grep jill /etc/passwd
jill:x:1001:1002::/home/jill:/bin/sh
root@debian-jessie:/core/examples# grep jill /etc/group
cfengineers:x:1001:jill
jill:x:1002:
It's important to remember we made no promise about the presence of
the cfengineers
group in the above example. We can see what would
happen when the cfengineers
group was not present.
root@debian-jessie:/core/examples# grep cfengineers /etc/group
root@debian-jessie:/core/examples# cf-agent -KIf ./local_user_secondary_group_member.cf
usermod: group 'cfengineers' does not exist
error: Command returned error while modifying user 'jill'. (Command line: '/usr/sbin/usermod -G "cfengineers" jill')
info: User promise not kept
Get a list of users
body common control
{
bundlesequence => { test };
}
bundle agent test
{
vars:
"allusers" slist => getusers("zenoss,mysql,at","12,0");
reports:
linux::
"Found user $(allusers)";
}
Tutorials
Familiarize yourself with CFEngine by following these step by step tutorials.
High availability
Overview
Although CFEngine is a distributed system, with decisions made by autonomous agents running on each node, the hub can be viewed as a single point of failure. In order to be able to play both roles that hub is responsible for - policy serving and report collection - High availability feature was introduced in 3.6.2. Essentially it is based on well known and broadly used cluster resource management tools - corosync and pacemaker as well as PostgreSQL streaming replication feature.
Design
CFEngine High availability is based on redundancy of all components, most importantly the PostgreSQL database. Active-passive PostgreSQL database configuration is the essential part of High Availability feature. While PostgreSQL supports different replication methods and active-passive configuration schemes, it doesn't provide out-of-the-box database failover-failback mechanism. To support that the well established cluster resources management solution based on the Linux-HA project was selected.
Overview of CFEngine High availability is shown in the diagram below.
One hub is the active hub, while the other serves the role of a passive hub and is a fully redundant instance of the active one. If the passive host determines the active host is down, it will be promoted to active and will start serving the Mission Portal, collect reports and serve policy.
Corosync and pacemaker
Corosync and pacemaker are well known and broadly used mechanisms supporting cluster resource management. For CFEngine hub needs those are configured so they are managing PostgreSQL database and one or more IP addresses shared over the nodes in the cluster. In the ideal configuration one link managed by corosync/pacemaker is dedicated for PostgreSQL streaming replication and one for accessing Mission Portal so that once failover happens the change of active-passive roles and failover transition is transparent for end user. They can still use the same shared IP address to log in to the Mission Portal or use against API queries.
PostgreSQL
For best performance, PostgreSQL streaming replication was selected as the database replication mode. It provides capability of shipping Write Ahead Log (WAL) entries from active server to all standby database servers. This is a PostgreSQL 9.0 and above feature allowing continuous recovery and almost immediate visibility of data inserted to primary server by the standby. For more information about PostgreSQL streaming replication please see PostgreSQL documentation.
CFEngine
In a High availability setup all the clients are aware of existence of more than one hub. Current active hub is selected as a policy server and policy fetching and report collection is done by the active hub. One of the differences comparing to single-hub installation is that instead of having one policy server, clients have a list of hubs where they should fetch policy and initiate report collection if using call collect. Also after bootstrapping to either active or passive hub clients are implicitly redirected to the active one. After that trust is established between the client and both active and passive hub so that all clients are capable to communicate with both. This allows transparent transition to the passive hub once fail-over is happening, as all the clients have already established trust with the passive hub as well.
Mission Portal
Mission Portal since 3.6.2 has a new indicator whitch shows the status of the High availability configuration.
High availability status is constantly monitored so that once some malfunction is discovered the user is notified about the degraded state of the system. Besides simple visualization of High Availability, the user is able to get detailed information regarding the reason for a degraded state, as well as when data was last reported from each hub. This gives quite comprehensive knowledge and overview of the whole setup.
Inventory
There are also new Mission Portal inventory variables indicating the IP address of the active hub instance and status of the High availability installation on each of the hubs. Looking at inventory reports is especially helpful to diagnose any problems when High availability is reported as degraded.
CFEngine High availability installation
Existing CFEngine Enterprise installations can upgrade their single-node hub to a High availability system in versions 3.6.2 and newer. Detailed instructions how to upgrade from single hub to High Availability or how to install CFEngine High availability from scratch can be found in the Installation guide.
Installation guide
Overview
This tutorial is describing the installation steps of the CFEngine High availability feature. It is suitable for both upgrading existing CFEngine installations to HA and for installing HA from scratch. Before starting installation we strongly recommend reading the CFEngine High availability overview.
Installation procedure
As with most High availability systems, setting it up requires carefully following a series of steps with dependencies on network components. The setup can therefore be error-prone, so if you are a CFEngine Enterprise customer we recommend that you contact support for assistance if you do not feel 100% comfortable of doing this on your own.
Please also make sure you have a valid license for the passive hub so that it will be able to handle all your CFEngine clients in case of failover.
Hardware configuration and OS pre-configuration steps
- CFEngine 3.15.3 (or later) hub package for RHEL7 or CentOS7.
- We recommend selecting dedicated interface used for PostgreSQL replication and optionally one for heartbeat.
- We recommend having one shared IP address assigned for interface where MP is accessible (optionally) and one where PostgreSQL replication is configured (mandatory).
- Both active and passive hub machines must be configured so that host names are different.
- Basic hostname resolution works (hub names can be placed in /etc/hosts or DNS configured).
Example configuration used in this tutorial
In this tutorial we use the following network configuration:
- Two nodes, one acting as active (node1) and one acting as passive (node2).
- Optionally a third node (node3) used as a database backup for offsite replication.
- Each node having three NICs so that eth0 is used for the heartbeat, eth1 is used for PostgreSQL replication and eth2 is used for MP and bootstrapping clients.
- IP addresses configured as follows:
Node | eth0 | eth1 | eth2 |
---|---|---|---|
node1 | 192.168.0.10 | 192.168.10.10 | 192.168.100.10 |
node2 | 192.168.0.11 | 192.168.10.11 | 192.168.100.11 |
node3 (optional) | --- | 192.168.10.12 | 192.168.100.12 |
cluster shared | --- | --- | 192.168.100.100 |
Detailed network configuration is shown on the picture below:
Install cluster management tools
On both nodes:
yum -y install pcs pacemaker cman fence-agents
In order to operate cluster, proper fencing must be configured but description how to fence cluster and what mechanism use is out of the scope of this document. For reference please use the Red Hat HA fencing guide.
IMPORTANT: please carefully follow the indicators describing if the given step should be performed on the active (node1), the passive (node2) or both nodes.
Make sure that the hostnames of all nodes nodes are node1 and node2 respectively. Running the command
uname -n | tr '[A-Z]' '[a-z]'
should return the correct node name. Make sure that the DNS or entries in /etc/hosts are updated so that hosts can be accessed using their host names.In order to use pcs to manage the cluster, create the hacluster user designated to manage the cluster with
passwd hacluster
on both nodes.Make sure that pcsd demon is started and configure both nodes so that it will be enabled to boot on startup on both nodes.
codeservice pcsd start chkconfig pcsd on
Authenticate hacluster user for each node of the cluster. Run the command below on the node1:
commandpcs cluster auth node1 node2 -u hacluster
After entering password, you should see a message similar to one below:
outputnode1: Authorized node2: Authorized
Create the cluster by running the following command on the node1:
commandpcs cluster setup --name cfcluster node1 node2
This will create the cluster
cfcluster
consisting of node1 and node2.Give the cluster time to settle (cca 1 minute) and then start the cluster by running the following command on the node1:
commandpcs cluster start --all
This will start the cluster and all the necessary deamons on both nodes.
At this point the cluster should be up and running. Running
pcs status
should print something similar to the output below.outputCluster name: cfcluster WARNING: no stonith devices and stonith-enabled is not false Stack: cman Current DC: node2 (version 1.1.18-3.el6-bfe4e80420) - partition with quorum Last updated: Wed Oct 17 12:25:42 2018 Last change: Wed Oct 17 12:24:52 2018 by root via crmd on node2 2 nodes configured 0 resources configured Online: [ node1 node2 ] No resources Daemon Status: cman: active/disabled corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled
If you are setting up just a testing environment without fencing, you should disable it now (**on the node1**):
codepcs property set stonith-enabled=false pcs property set no-quorum-policy=ignore
Before the PostgreSQL replication is setup, we need to set up a floating IP address that will always point to the active node and configure some basic resource parameters (**on the node1**):
codepcs resource defaults resource-stickiness="INFINITY" pcs resource defaults migration-threshold="1" pcs resource create cfvirtip IPaddr2 ip=192.168.100.100 cidr_netmask=24 --group cfengine pcs cluster enable --all
Verify that the cfvirtip resource is properly configured and running.
commandpcs status
should give something like this:
outputCluster name: cfcluster Last updated: Tue Jul 7 09:29:10 2015 Last change: Fri Jul 3 08:41:24 2015 Stack: cman Current DC: node1 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured 1 Resources configured Online: [ node1 node2 ] Full list of resources: Resource Group: cfengine cfvirtip (ocf::heartbeat:IPaddr2): Started node1
PostgreSQL configuration
- Install the CFEngine hub package on both node1 and node2.
Make sure CFEngine is not running (**on both node1 and node2**):
commandservice cfengine3 stop
Configure PostgreSQL on node1:
Create two special directories owned by the cfpostgres user:
codemkdir -p /var/cfengine/state/pg/{data/pg_arch,tmp} chown -R cfpostgres:cfpostgres /var/cfengine/state/pg/{data/pg_arch,tmp}
Modify the /var/cfengine/state/pg/data/postgresql.conf configuration file to set the following options accordingly (**uncomment the lines if they are commented out**):
codelisten_addresses = '*' wal_level = replica max_wal_senders = 5 wal_keep_segments = 16 hot_standby = on restart_after_crash = off archive_mode = on archive_command = 'cp %p /var/cfengine/state/pg/data/pg_arch/%f'
Modify the pg_hba.conf configuration file to enable access to PostgreSQL for replication between the nodes (note that the second pair of IP addresses, not the heartbeat pair, is used here):
codeecho "host replication all 192.168.100.10/32 trust" >> /var/cfengine/state/pg/data/pg_hba.conf echo "host replication all 192.168.100.11/32 trust" >> /var/cfengine/state/pg/data/pg_hba.conf
IMPORTANT: The above configuration allows accessing PostgreSQL without any authentication from both cluster nodes. For security reasons we strongly advise to create a replication user in PostgreSQL and protect access using a password or certificate. Furthermore, we advise using ssl-secured replication instead of the unencrypted method described here if the hubs are in an untrusted network.
Do an initial sync of PostgreSQL:
Start PostgreSQL on node1:
commandpushd /tmp; su cfpostgres -c "/var/cfengine/bin/pg_ctl -w -D /var/cfengine/state/pg/data -l /var/log/postgresql.log start"; popd
On node2, initialize PostgreSQL from node1 (again using the second IP, not the heartbeat IP):
coderm -rf /var/cfengine/state/pg/data/* pushd /tmp; su cfpostgres -c "/var/cfengine/bin/pg_basebackup -h 192.168.10.10 -U cfpostgres -D /var/cfengine/state/pg/data -X stream -P"; popd
On node2, create the standby.conf file and configure PostgreSQL to run as a hot-standby replica:
codecat <<EOF > /var/cfengine/state/pg/data/standby.conf #192.168.100.100 is the shared over cluster IP address of active/master cluster node primary_conninfo = 'host=192.168.100.100 port=5432 user=cfpostgres application_name=node2' restore_command = 'cp /var/cfengine/state/pg/pg_arch/%f %p' EOF chown --reference /var/cfengine/state/pg/data/postgresql.conf /var/cfengine/state/pg/data/standby.conf echo "include 'standby.conf'" >> /var/cfengine/state/pg/data/postgresql.conf touch /var/cfengine/state/pg/data/standby.signal
Start PostgreSQL on the node2 by running the following command:
commandpushd /tmp; su cfpostgres -c "/var/cfengine/bin/pg_ctl -D /var/cfengine/state/pg/data -l /var/log/postgresql.log start"; popd
Check that PostgreSQL replication is setup and working properly:
The node2 should report it is in the recovery mode:
command/var/cfengine/bin/psql -x cfdb -c "SELECT pg_is_in_recovery();"
should return:
output-[ RECORD 1 ]-----+-- pg_is_in_recovery | t
The node1 should report it is replicating to node2:
command/var/cfengine/bin/psql -x cfdb -c "SELECT * FROM pg_stat_replication;"
should return something like this:
output-[ RECORD 1 ]----+------------------------------ pid | 11401 usesysid | 10 usename | cfpostgres application_name | node2 client_addr | 192.168.100.11 client_hostname | node2-pg client_port | 33958 backend_start | 2018-10-16 14:19:04.226773+00 backend_xmin | state | streaming sent_lsn | 0/61E2C88 write_lsn | 0/61E2C88 flush_lsn | 0/61E2C88 replay_lsn | 0/61E2C88 write_lag | flush_lag | replay_lag | sync_priority | 0 sync_state | async
Stop PostgreSQL on both nodes:
commandpushd /tmp; su cfpostgres -c "/var/cfengine/bin/pg_ctl -D /var/cfengine/state/pg/data -l /var/log/postgresql.log stop"; popd
Remove the hot-standby configuration on node2. It will be handled by the cluster resource and the resource agent.
coderm -f /var/cfengine/state/pg/data/standby.signal rm -f /var/cfengine/state/pg/data/standby.conf sed -i "/standby\.conf/d" /var/cfengine/state/pg/data/postgresql.conf
Cluster resource configuration
Download the PostgreSQL resource agent supporting the CFEngine HA setup on both nodes.
codewget https://raw.githubusercontent.com/cfengine/core/master/contrib/pgsql_RA /bin/cp pgsql_RA /usr/lib/ocf/resource.d/heartbeat/pgsql chown --reference /usr/lib/ocf/resource.d/heartbeat/{IPaddr2,pgsql} chmod --reference /usr/lib/ocf/resource.d/heartbeat/{IPaddr2,pgsql}
Create the PostgreSQL resource (**on node1**).
codepcs resource create cfpgsql pgsql \ pgctl="/var/cfengine/bin/pg_ctl" \ psql="/var/cfengine/bin/psql" \ pgdata="/var/cfengine/state/pg/data" \ pgdb="cfdb" pgdba="cfpostgres" repuser="cfpostgres" \ tmpdir="/var/cfengine/state/pg/tmp" \ rep_mode="async" node_list="node1 node2" \ primary_conninfo_opt="keepalives_idle=60 keepalives_interval=5 keepalives_count=5" \ master_ip="192.168.100.100" restart_on_promote="true" \ logfile="/var/log/postgresql.log" \ config="/var/cfengine/state/pg/data/postgresql.conf" \ check_wal_receiver=true restore_command="cp /var/cfengine/state/pg/data/pg_arch/%f %p" \ op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \ op monitor timeout="60s" interval="4s" on-fail="restart" --disable
Configure PostgreSQL to work in Master/Slave (active/standby) mode (**on node1**).
commandpcs resource master mscfpgsql cfpgsql master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
Tie the previously configured shared IP address and PostgreSQL cluster resources to make sure both will always run on the same host and add migration rules to make sure that resources will be started and stopped in the correct order (**on node1**).
codepcs constraint colocation add cfengine with Master mscfpgsql INFINITY pcs constraint order promote mscfpgsql then start cfengine symmetrical=false score=INFINITY pcs constraint order demote mscfpgsql then stop cfengine symmetrical=false score=0
Enable and start the new resource now that it is fully configured (**on node1**).
commandpcs resource enable mscfpgsql --wait=30
Verify that the constraints configuration is correct.
commandpcs constraint
should give:
outputLocation Constraints: Resource: mscfpgsql Enabled on: node1 (score:INFINITY) (role: Master) Ordering Constraints: promote mscfpgsql then start cfengine (score:INFINITY) (non-symmetrical) demote mscfpgsql then stop cfengine (score:0) (non-symmetrical) Colocation Constraints: cfengine with mscfpgsql (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master)
Verify that the cluster is now fully setup and running.
commandcrm_mon -Afr1
should give something like:
outputStack: cman Current DC: node1 (version 1.1.18-3.el6-bfe4e80420) - partition with quorum Last updated: Tue Oct 16 14:19:37 2018 Last change: Tue Oct 16 14:19:04 2018 by root via crm_attribute on node1 2 nodes configured 3 resources configured Online: [ node1 node2 ] Full list of resources: Resource Group: cfengine cfvirtip (ocf::heartbeat:IPaddr2): Started node1 Master/Slave Set: mscfpgsql [cfpgsql] Masters: [ node1 ] Slaves: [ node2 ] Node Attributes: * Node node1: + cfpgsql-data-status : LATEST + cfpgsql-master-baseline : 0000000004000098 + cfpgsql-receiver-status : normal (master) + cfpgsql-status : PRI + master-cfpgsql : 1000 * Node node2: + cfpgsql-data-status : STREAMING|ASYNC + cfpgsql-receiver-status : normal + cfpgsql-status : HS:async + master-cfpgsql : 100
IMPORTANT: Please make sure that there's one Master node and one Slave node and that the cfpgsql-status for the active node is reported as PRI and passive as HS:async or HS:alone.
CFEngine configuration
Create the HA configuration file on both nodes.
codecat <<EOF > /var/cfengine/ha.cfg cmp_master: PRI cmp_slave: HS:async,HS:sync,HS:alone cmd: /usr/sbin/crm_attribute -l reboot -n cfpgsql-status -G -q EOF
Mask the cf-postgres.service and make sure it is not required by the cf-hub.service on both nodes (PostgreSQL is managed by the cluster resource, not by the service).
codesed -ri s/Requires/Wants/ /usr/lib/systemd/system/cf-hub.service systemctl daemon-reload systemctl mask cf-postgres.service
Bootstrap the nodes.
Bootstrap the node1 to itself and make sure the initial policy (
promises.cf
) evaluation is skipped:commandcf-agent --bootstrap 192.168.100.10 --skip-bootstrap-policy-run
Bootstrap the node2 to node1 (to establish trust) and then to itself, again skipping the initial policy evaluation:
codecf-agent --bootstrap 192.168.100.10 --skip-bootstrap-policy-run cf-agent --bootstrap 192.168.100.11 --skip-bootstrap-policy-run
Stop CFEngine on both nodes.
commandservice cfengine3 stop
Create the HA JSON configuration file on both nodes.
codecat <<EOF > /var/cfengine/masterfiles/cfe_internal/enterprise/ha/ha_info.json { "192.168.100.10": { "sha": "@NODE1_PKSHA@", "internal_ip": "192.168.100.10" }, "192.168.100.11": { "sha": "@NODE2_PKSHA@", "internal_ip": "192.168.100.11" } } EOF
The
@NODE1_PKSHA@
and@NODE2_PKSHA@
strings are placeholders for the host key hashes of the nodes. Replace the placeholders with real values obtained by (on any node):commandcf-key -s
IMPORTANT: Copy over only the hashes, without the
SHA=
prefix.On both nodes, add the following class definition to the /var/cfengine/masterfiles/def.json file to enable HA:
def.json{ "classes": { "enable_cfengine_enterprise_hub_ha": [ "any::" ] } }
On both nodes, run
cf-agent -Kf update.cf
to make sure that the new policy is copied from masterfiles to inputs.Start CFEngine on both nodes.
commandservice cfengine3 start
Check that the CFEngine HA setup is working by logging in to the Mission Portal at the https://192.168.100.100 address in your browser. Note that it takes up to 15 minutes for everything to settle and the
OK
HA status being reported in the Mission Portal's header.
Configuring 3rd node as disaster-recovery or database backup (optional)
Install the CFEngine hub package on the node which will be used as disaster-recovery or database backup node (node3).
Bootstrap the disaster-recovery node to active node first (establish trust between hubs) and then bootstrap it to itself. At this point hub will be capable of collecting reports and serve policy.
Stop cf-execd and cf-hub processes.
Make sure that PostgreSQL configuration allows database replication connection from 3rd node (see PostgreSQL configuration section, point 5.3 for more details).
Repeat steps 4 - 6 from PostgreSQL configuration to enable and verify database replication connection from the node3. Make sure that both the node2 and node3 are connected to active database node and streaming replication is in progress.
Running the following command on node1:
command/var/cfengine/bin/psql cfdb -c "SELECT * FROM pg_stat_replication;"
Should give:
outputpid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | state | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state ------+----------+------------+------------------+----------------+-----------------+-------------+-------------------------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------ 9252 | 10 | cfpostgres | node2 | 192.168.100.11 | | 58919 | 2015-08-24 07:14:45.925341+00 | streaming | 0/2A7034D0 | 0/2A7034D0 | 0/2A7034D0 | 0/2A7034D0 | 0 | async 9276 | 10 | cfpostgres | node3 | 192.168.100.12 | | 52202 | 2015-08-24 07:14:46.038676+00 | streaming | 0/2A7034D0 | 0/2A7034D0 | 0/2A7034D0 | 0/2A7034D0 | 0 | async (2 rows)
Modify HA JSON configuration file to contain information about the node3 (see CFEngine configuration, step 2). You should have configuration similar to one below:
commandcat /var/cfengine/masterfiles/cfe_internal/enterprise/ha/ha_info.json
output{ "192.168.100.10": { "sha": "b1463b08a89de98793d45a52da63d3f100247623ea5e7ad5688b9d0b8104383f", "internal_ip": "192.168.100.10", "is_in_cluster" : true, }, "192.168.100.11": { "sha": "b13db51615afa409a22506e2b98006793c1b0a436b601b094be4ee4b32b321d5", "internal_ip": "192.168.100.11", }, "192.168.100.12": { "sha": "98f14786389b2fe5a93dc3ef4c3c973ef7832279aa925df324f40697b332614c", "internal_ip": "192.168.100.12", "is_in_cluster" : false, } }
Please note that
is_in_cluster
parameter is optional for the 2 nodes in the HA cluster and by default is set to true. For the 3-node setup, the node3, which is not part of the cluster, MUST be marked with"is_in_cluster" : false
configuration parameter.Start the cf-execd process (don't start cf-hub process as this is not needed while manual failover to the node3 is not performed). Please also note that during normal operations the cf-hub process should not be running on the node3.
Manual failover to disaster-recovery node
Before starting manual failover process make sure both active and passive nodes are not running.
Verify that PostgreSQL is running on 3rd node and data replication from active node is not in progress. If database is actively replicating data with active cluster node make sure that this process will be finished and no new data will be stored in active database instance.
After verifying that replication is finished and data is synchronized between active database node and replica node (or once node1 and node2 are both down) promote PostgreSQL to exit recovery and begin read-write operations
cd /tmp && su cfpostgres -c "/var/cfengine/bin/pg_ctl -c -w -D /var/cfengine/state/pg/data -l /var/log/postgresql.log promote"
.In order to make failover process as easy as possible there is
"failover_to_replication_node_enabled"
class defined both in /var/cfengine/masterfiles/controls/VERSION/def.cf and /var/cfengine/masterfiles/controls/VERSION/update_def.cf. In order to stat collecting reports and serving policy from 3rd node uncomment the line defining mentioned class.
IMPORTANT: Please note that as long as any of the active or passive cluster nodes is accessible by client to be contacted, failover to 3rd node is not possible. If the active or passive node is running and failover to 3rd node is required make sure to disable network interfaces where clients are bootstrapped to so that clients won't be able to access any other node than disaster-recovery.
Troubleshooting
If either the IPaddr2 or pgslq resource is not running, try to enable it first with
pcs cluster enable --all
. If this is not strting the resources, you can try to run them in debug mode with this commandpcs resource debug-start <resource-name>
. The latter command should print diagnostics messages on why resources are not started.If
crm_mon -Afr1
is printing errors similar to the belowcommandpcs status
outputCluster name: cfcluster Last updated: Tue Jul 7 11:27:23 2015 Last change: Tue Jul 7 11:02:40 2015 Stack: cman Current DC: node1 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured 3 Resources configured Online: [ node1 ] OFFLINE: [ node2 ] Full list of resources: Resource Group: cfengine cfvirtip (ocf::heartbeat:IPaddr2): Started node1 Master/Slave Set: mscfpgsql [cfpgsql] Stopped: [ node1 node2 ] Failed actions: cfpgsql_start_0 on node1 'unknown error' (1): call=13, status=complete, last-rc-change='Tue Jul 7 11:25:32 2015', queued=1ms, exec=137ms
You can try to clear the errors by running
pcs resource cleanup <resource-name>
. This should clean errors for the appropriate resource and make the cluster restart it.commandpcs resource cleanup cfpgsql
outputResource: cfpgsql successfully cleaned up
commandpcs status
outputCluster name: cfcluster Last updated: Tue Jul 7 11:29:36 2015 Last change: Tue Jul 7 11:29:08 2015 Stack: cman Current DC: node1 - partition with quorum Version: 1.1.11-97629de 2 Nodes configured 3 Resources configured Online: [ node1 ] OFFLINE: [ node2 ] Full list of resources: Resource Group: cfengine cfvirtip (ocf::heartbeat:IPaddr2): Started node1 Master/Slave Set: mscfpgsql [cfpgsql] Masters: [ node1 ] Stopped: [ node2 ]
After cluster crash make sure to always start the node that should be active first, and then the one that should be passive. If the cluster is not running on the given node after restart you can enable it by running the following command:
commandpcs cluster start
outputStarting Cluster...
JSON and YAML support in CFEngine
Introduction
JSON is a well-known data language. It even has a specification (See http://json.org).
YAML is another well-known data language. It has a longer, much more complex specification (See http://yaml.org).
CFEngine has core support for JSON and YAML. Let's see what it can do.
Problem statement
We'd like to read, access, and merge JSON-sourced data structures: they should be weakly typed, arbitrarily nested, with consistent quoting and syntax.
We'd like to read, access, and merge YAML-sourced data structures just like JSON-sourced, to keep policy and internals simple.
In addition, we must not break backward compatibility with CFEngine
3.5 and older, so we'd like to use the standard CFEngine array a[b]
syntax.
Data containers
A new data type, the data container, was introduced in 3.6.
It's simply called data
. The documentation with some examples is at https://cfengine.com/docs/master/reference-promise-types-vars.html#data-container-variables
Reading JSON
There are many ways to read JSON data; here are a few:
readjson()
: read from a JSON file, e.g."mydata" data => readjson("/my/file", 100k);
parsejson()
: read from a JSON string, e.g."mydata" data => parsejson('{ "x": "y" }');
data_readstringarray()
anddata_readstringarrayidx()
: read text data from a file, split it on a delimiter, and make them into structured data.mergedata()
: merge data containers, slists, and classic CFEngine arrays, e.g."mydata" data => mergedata(container1, slist2, array3);
mergedata
in particular is very powerful. It can convert a slist or a classic CFEngine array to a data container easily: "mydata" data => mergedata(myslist);
Reading YAML
There are two ways to read YAML data:
readyaml()
: read from a YAML file, e.g."mydata" data => readyaml("/my/file.yaml", 100k);
parseyaml()
: read from a YAML string, e.g."mydata" data => parseyaml('- arrayentry1');
Since these functions return data containers, everything about JSON-sourced data structures applies to YAML-sourced data structures as well.
Accessing JSON
To access JSON data, you can use:
- the
nth()
function to access an array element, e.g."myx" string => nth(container1, 0);
- the
nth
function to access a map element, e.g."myx" string => nth(container1, "x");
- the
a[b]
notation, e.g."myx" string => "$(container1[x])";
. You can nest, e.g.a[b][c][0][d]
. This only works if the element is something that can be expanded in a string. So a number or a string work. A list of strings or numbers works. A key-value map underx
won't work. - the
getindices()
andgetvalues()
functions, just like classic CFEngine arrays
A full example
This example can be saved and run. It will load a key-value map where the keys are class names and the values are hostname regular expressions or class names.
- if your host name is
c
orb
or the classesc
orb
are defined, thedev
class will be defined - if your host name is
flea
or the classflea
is defined, theprod
class will be defined - if your host name is
a
or the classa
is defined, theqa
class will be defined - if your host name is
linux
or the classlinux
is defined, theprivate
class will be defined
Easy, right?
body common control
{
bundlesequence => { "run" };
}
bundle agent run
{
vars:
"bykey" data => parsejson('{ "dev": ["c", "b"], "prod": ["flea"], "qa": ["a"], "private": ["linux"] }');
"keys" slist => getindices("bykey");
classes:
# define the class from the key name if any of the items under the key match the host name
"$(keys)" expression => regcmp("$(bykey[$(keys)])", $(sys.host));
# define the class from the key name if any of the items under the key are a defined class
"$(keys)" expression => classmatch("$(bykey[$(keys)])");
reports:
"keys = $(keys)";
"I am in class $(keys)" if => $(keys);
}
So, where's the magic? Well, if you're familiar with classic CFEngine
arrays, you will be happy to hear that the exact same syntax works
with them. In other words, data containers don't change how you use
CFEngine. You still use getindices
to get the keys, then iterate
through them and look up values.
Well, you can change
"bykey" data => parsejson('{ "dev": ["c", "b"], "prod": ["flea"], "qa": ["a"], "private": ["linux"] }');
with
"bykey" data => data_readstringarray(...);
and read the same container from a text file. The file should be formatted like this to produce the same data as above:
dev c b
prod flea
qa a
private linux
You can also use
"bykey" data => readjson(...);
and read the same container from a JSON file.
Summary
Using JSON and YAML from CFEngine is easy and does not change how you use CFEngine. Try it out and see for yourself!
Installing CFEngine Enterprise agent
This is the full version of CFEngine Enterprise host, but the number of hosts is limited to 25.
NOTE: Please make sure to have installed the CFEngine Policyserver before moving on to install the hosts.
System requirements
CFEngine Hosts (clients)
- 32/64-bit machines with a recent version of Linux
- 20 mb of memory
- 20mb of disk space
- Port 5308 needs to be open
The installation script below has been tested on Red Hat, CentOS, SUSE, Debian and Ubuntu.
- Download and Install CFEngine Host Run the following command to download and automatically install CFEngine on a 32-bit or 64-bit Linux machine (the script will detect correct flavor and architecture).
wget https://s3.amazonaws.com/cfengine.packages/quick-install-cfengine-enterprise.sh && sudo bash ./quick-install-cfengine-enterprise.sh agent
- Bootstrap the Host Once installed, the host needs to bootstrap to your CFEngine policy server.
sudo /var/cfengine/bin/cf-agent --bootstrap <Name or IP address of policy server>
If you encounter any issue, please make sure the host is on the same domain/subnet as CFEngine policy server will only allow connection from these trusted sources as default configuration.
- Congratulation you are done! The CFEngine host is installed and ready. That was easy, wasn't it?
If you would like to see what version of CFEngine you are running, type:
/var/cfengine/bin/cf-promises --version
Now, you have a client-server CFEngine running. If you would like to install more hosts, simply repeat steps 1 to 3 above. You are free to have up to 25 hosts. Enjoy!
Once you have installed the number of hosts you want, a good next step would be to take a look at our How-to write your first policy tutorial.
Managing local users
In this tutorial we will show how to use CFEngine to manage users, add them to
groups, setup their home directory and copy ssh-keys to their ~/.ssh
directory
as part of creating the user.
- Create some files and groups that we will use
Create the files id_rsa
and id_rsa.pub
in /tmp
.
touch /tmp/id_rsa /tmp/id_rsa.pub
Create user group security and webadmin.
# sudo groupadd security
# sudo groupadd webadmin
- Create CFEngine policy called
users.cf
Create a file /tmp/users.cf
with the following content:
body common control
{
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent main
{
vars:
"users" slist => { "adam", "eva" };
users:
"$(users)"
policy => "present",
home_dir => "/home/$(users)",
group_primary => "users",
groups_secondary => { "security", "webadmin" },
shell => "/bin/bash/",
home_bundle => setup_home_dir("$(users)");
}
bundle agent setup_home_dir(user)
{
vars:
"keys" slist => { "id_rsa", "id_rsa.pub" };
files:
"/home/$(user)/." create => "true";
"/home/$(user)/.ssh/." create => "true";
"/home/$(user)/.ssh/$(keys)" copy_from => local_cp("/tmp/$(keys)");
}
- Test it out, and verify the result
Run CFEngine:
/var/cfengine/bin/cf-agent -fK /tmp/users.cf
Verify the result: Have users have been created?
grep -P "adam|eva" /etc/passwd
Congratulations! You should now see the users adam and eva listed.
Verify the result: Have users home directory have been created?
ls /home | grep -P "adam|eva"
Congratulations! You should now see adam and eva listed.
Verify the result: Have users have been added to the correct groups?
grep -P "adam|eva" /etc/group
Congratulations! You should now see adam and eva added to the groups security and webadmin. NOTE: CFEngine's users type promise will not create groups, so you must make sure the groups exists.
Verify the result: Have ssh-keys have been copied from /tmp
to user's ~/.ssh
directory?
ls /home/adam/.ssh /home/eva/.ssh
Congratulations! You should now see the files id_rsa
and id_rsa.pub
.
Ps. If you would like play around with the policy, delete the users after each run with the command
deluser -r username
Mission accomplished!
Managing network time protocol
In this tutorial we will write a simple policy to ensure that the latest version of the NTP service is installed on your system. Once the NTP software is installed, we will extend the policy to manage the service state as well as the software configuration.
Note: For simplicity, in this tutorial we will work directly on top of the Masterfiles Policy Framework (MPF) in /var/cfengine/masterfiles
(*masterfiles*) and we will not use version control.
Ensuring the NTP package is installed
bundle agent ntp
{
vars:
"ntp_package_name" string => "ntp";
packages:
"$(ntp_package_name)" -> { "StandardsDoc 3.2.1" }
policy => "present",
handle => "ntp_packages_$(ntp_package_name)",
classes => results("bundle", "ntp_package");
}
What does this policy do?
Let's walk through the different sections of the code do see how it works.
bundle agent ntp
You can think of bundles as a collection of desired states. You can have as many bundles as you would like, and also reference them from within themselves. In fact, they are somewhat similar to function calls in other programming languages. Let's dive deeper into the code in the ntp bundle.
vars
vars:
vars
is a promise type that ensures the presence of variables that hold specific values. vars:
starts a promise type block which ends when the next promise type block is declared.
ntp_package_name
"ntp_package_name" string => "ntp";
A variable with the name ntp_package_name
is declared and it is assigned a value, ntp
. This string variable will be referenced in the other sections of the bundle.
packages
packages:
"$(ntp_package_name)" -> { "StandardsDoc 3.2.1" }
policy => "present",
handle => "ntp_packages_$(ntp_package_name)",
classes => results("bundle", "ntp_package");
packages
is a promise type that ensures the presence or absence of a package on a system.
$(ntp_package_name)
"$(ntp_package_name)" -> { "StandardsDoc 3.2.1" }
Notice the ntp_package_name
variable is referenced here, which evaluates to ntp
as the promiser. You can also associate a stakeholder aka promisee to this promiser. The stakeholder association is optional, but is particularly useful in when you wish to provide some structure in your policy to tie it to a business rule. In this example, what we are stating is this - "Make sure NTP is installed as it is described in StandardsDoc 3.2.1".
This promiser has a number of additional attributes defined:
policy
policy => "present",
The package_policy attribute describes what you want to do the package. In this case you want to ensure that it is present on the system. Other valid values of this attribute include delete, update, patch, reinstall, addupdate, and verify. Because of the self-healing capabilities of CFEngine, the agents will continuously check to make sure the package is installed. If it is not installed, CFEngine will try to install it according to its policy.
handle
handle => "ntp_packages_$(ntp_package_name)",
The handle uniquely identifies a promise within a policy. A recommended naming scheme for the handle is bundle_name_promise_type_class_restriction_promiser
. It is often used for documentation and compliance purposes. As shown in this example, you can easily substitute values of variables for the handle.
classes
classes => results("bundle", "ntp_package_");
classes
provide context which can help drive the logic in your policies. In this example, classes for each promise outcome are defined prefixed with ntp_package_
, for details check out the implementation of body classes results
in the stdlib. For example, ntp_package_repaired
will be defined if cf-agent did not have the ntp package installed and had to install it. ntp_package_kept
would be defined if the ntp package is already installed and ntp_package_notkept
would be defined.
On your hub create services/ntp.cf
inside masterfiles with the following content:
bundle agent ntp
{
vars:
"ntp_package_name" string => "ntp";
packages:
"$(ntp_package_name)" -> { "StandardsDoc 3.2.1" }
policy => "present",
handle => "ntp_packages_$(ntp_package_name)",
classes => results("bundle", "ntp_package");
}
Now, check the syntax, it's always a good idea any time you edit policy.
[root@hub masterfiles]# cf-promises -f ./services/ntp.cf
[root@hub masterfiles]# echo $?
0
Now, we need to make sure the agent knows it should use this policy file and bundle. Create def.json
an Augments file with the following content:
{
"inputs": [ "services/ntp.cf" ],
"vars": {
"control_common_bundlesequence_end": [ "ntp" ]
}
}
Validate it.
python -m json.tool < def.json
{
"inputs": [
"services/ntp.cf"
],
"vars": {
"control_common_bundlesequence_end": [
"ntp"
]
}
}
Force a policy update. Remember, CFEngine is running in the background, so it's possible that by the time you force a policy update and run the agent may have already done it and your output may differ.
cf-agent -KIf update.cf
In the output, you should see something like:
info: Updated '/var/cfengine/inputs/services/ntp.cf' from source '/var/cfengine/masterfiles/services/ntp.cf' on 'localhost'
Now force a policy run.
cf-agent -KI
info: Successfully installed package 'ntp'
Now that we have successfully promised the package, let's move on to the service.
Manage NTP service
Now we will extend the policy to ensure that the NTP service is running.
Now that the NTP service has been installed on the system, we need to make sure that it is running.
bundle agent ntp
{
vars:
"ntp_package_name" string => "ntp";
redhat::
"ntp_service_name" string => "ntpd";
debian::
"ntp_service_name" string => "ntp";
packages:
"$(ntp_package_name)" -> { "StandardsDoc 3.2.1" }
policy => "present",
handle => "ntp_packages_$(ntp_package_name)",
classes => results("bundle", "ntp_package");
services:
"$(ntp_service_name)" -> { "StandardsDoc 3.2.2" }
service_policy => "start",
classes => results( "bundle", "ntp_service")
reports:
ntp_service_repaired.inform_mode::
"NTP service repaired";
}
What does this policy do?
Let's dissect this policy and review the differences in the policy.
vars
redhat::
"ntp_service_name" string => "ntpd";
debian::
"ntp_service_name" string => "ntp";
The first thing that you will notice is that the variable declarations section has been expanded. Recall that you completed part 1 of this tutorial by creating packages promises that works across Debian and redhat. While the package name for NTP is the same between Debian and Red Hat, the service names are actually different. Therefore, classes introduced here to distinguish the service name for NTP between these two environments. The CFEngine agents automatically discover environment properties and defines hard classes that can be used - this includes classes such as debian
and redhat
that define the host's operating system.
reports
reports:
ntp_service_repaired.inform_mode::
"NTP service repaired";
The reports promise type emits information from the agent. Most commonly and by default, information is emitted to standard out. Reports are useful when tracking or reporting on the progress of the policy execution.
ntp_service_repaired.inform_mode::
This line restricts the context for the promises that follow to hosts that have ntp_service_repaired
and inform_mode
defined. Note: inform_mode
is defined when information level logging is requested, e.g. the -I
, --inform
, or --log-level inform
options are given to cf-agent
defined.
"NTP service repaired";
This defines the line that should be emitted by the reports
promise type.
Messages printed to standard out from reports promises are prefixed with the letter R
to distinguish them from other output.
R: NTP service repaired
Modify and run the policy
On your hub modify services/ntp.cf
introducing the new promises as described previously.
After making changes it's always a good idea to validate the policy file you modified, as well as the full policy set:
[root@hub masterfiles]# cf-promises -KI -f ./services/ntp.cf
[root@hub masterfiles]# cf-promises -KI -f ./promises.cf
If the code has no syntax error, you should see no output.
Perform a manual policy run and review the output to ensure that the policy executed successfully. Upon a successful run you should expect to see an output similar to this (depending on the init system your OS is using):
cf-agent -KIf update.cf ; cf-agent -KI
info: Copied file '/var/cfengine/masterfiles/services/ntp.cf' to '/var/cfengine/inputs/services/ntp.cf.cfnew' (mode '600')
info: Executing 'no timeout' ... '/sbin/chkconfig ntpd on'
info: Command related to promiser '/sbin/chkconfig ntpd on' returned code defined as promise kept 0
info: Completed execution of '/sbin/chkconfig ntpd on'
info: Executing 'no timeout' ... '/etc/init.d/ntpd start'
info: Completed execution of '/etc/init.d/ntpd start'
R: NTP service repaired
You have now written a complete policy to ensure that the NTP package is installed, and that the service is up and running.
Manage NTP configuration
Now we will manage the configuration file using the built-in mustache templating engine, set up appropriate file permissions, and restart the service when necessary.
By default, the NTP service leverages configuration properties specified in /etc/ntp.conf. In this tutorial, we introduce the concept of the files promise type. With this promise type, you can create, delete, and edit files using CFEngine policies. The example policy below illustrates the use of the files promise.
bundle agent ntp
{
vars:
linux::
"ntp_package_name" string => "ntp";
"config_file" string => "/etc/ntp.conf";
"driftfile" string => "/var/lib/ntp/drift";
"servers" slist => { "time.nist.gov" };
# For brevity, and since the template is small, we define it in-line
"config_template_string"
string => "# NTP Config managed by CFEngine
driftfile {{{driftfile}}}
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
{{#servers}}
server {{{.}}} iburst
{{/servers}}
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
";
redhat::
"ntp_service_name" string => "ntpd";
debian::
"ntp_service_name" string => "ntp";
packages:
"$(ntp_package_name)" -> { "StandardsDoc 3.2.1" }
policy => "present",
handle => "ntp_packages_$(ntp_package_name)",
classes => results("bundle", "ntp_package");
files:
"$(config_file)"
create => "true",
handle => "ntp_files_conf",
perms => mog( "644", "root", "root" ),
template_method => "inline_mustache",
edit_template_string => "$(config_template_string)",
template_data => mergedata( '{ "driftfile": "$(driftfile)", "servers": servers }' ),
classes => results( "bundle", "ntp_config" );
services:
"$(ntp_service_name)" -> { "StandardsDoc 3.2.2" }
service_policy => "start",
classes => results( "bundle", "ntp_service_running" );
ntp_config_repaired::
"$(ntp_service_name)" -> { "StandardsDoc 3.2.2" }
service_policy => "restart",
classes => results( "bundle", "ntp_service_config_change" );
reports:
ntp_service_running_repaired.inform_mode::
"NTP service started";
ntp_service_config_change_repaired.inform_mode::
"NTP service restarted after configuration change";
}
What does this policy do?
Let's review the different sections of the code, starting with the variable declarations which makes use of operating system environment for classification of the time servers.
vars
vars:
linux::
"ntp_package_name" string => "ntp";
"config_file" string => "/etc/ntp.conf";
"driftfile" string => "/var/lib/ntp/drift";
"servers" slist => { "utcnist.colorado.edu", "utcnist2.colorado.edu" };
# For brevity, and since the template is small, we define it in-line
"config_template_string"
string => "# NTP Config managed by CFEngine
driftfile {{{driftfile}}}
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
{{#servers}}
server {{{.}}} iburst
{{/servers}}
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
";
A few new variables are defined. The variables ntp_package_name
, config_file
, driftfile
, servers
, and config_template_string
are defined under the linux
context (so only linux hosts will define them). config_file
is the path to the ntp configuration file, driftfile
and servers
are both variables that will be used when rendering the configuration file and config_template_string
is the template that will be used to render the configuration file. While both driftfile
and servers
are set the same for all linux hosts, those variables could easily be set to different values under different contexts.
files
Now let's walk through the files promise in detail.
files:
"$(config_file)"
create => "true",
handle => "ntp_files_conf",
perms => mog( "644", "root", "root" ),
template_method => "inline_mustache",
edit_template_string => "$(config_template_string)",
template_data => mergedata( '{ "driftfile": "$(driftfile)", "servers": servers }' ),
classes => results( "bundle", "ntp_config" );
The promiser here is referenced by the config_file
variable. In this case, it is the configuration file for the NTP service. There are a number of additional attributes that describe this promise.
create
create => "true",
Valid values for this attribute are true
or false
to instruct the agent whether or not to create the file. In other words, the file must exist. If it does not exist, it will be created.
perms
perms => mog( "644", "root", "root" ),
This attribute sets the permissions and ownership of the file. mog()
is a perms
body in the CFEngine standard library that sets the mode
, owner
, and group
of the file. In this example, the permissions for the NTP configuration file are set to 644
with owner and group both assigned to root
.
handle
handle => "ntp_files_conf",
A handle uniquely identifies a promise within a policy set. The policy style guide recommends a naming scheme for the handles e.g. bundle_name_promise_type_class_restriction_promiser
. Handles are optional, but can be very useful when reviewing logs and can also be used to influence promise ordering with depends_on
.
classes
classes => results( "bundle", "ntp_config" );
The classes attribute here uses the results()
classes body from the standard library. The results()
body defines classes for every outcome a promise has. Every time this promise is executed classes will be defined bundle scoped classes prefixed with ntp_config
. If the promise changes the file content or permissions the class ntp_config_repaired
will be set.
template_method
template_method => "inline_mustache",
CFEngine supports multiple templating engines, the template_method attribute specifies how the promised file content will be resolved. The value inline_mustache
indicates that we will use the mustache templating engine and specify the template in-line, instead of in an external file.
edit_template_string
edit_template_string => "$(config_template_string)",
The edit_template_string
attribute is set to $(config_template_string)
which holds the mustache template used to render the file content.
template_data
template_data => mergedata( '{ "driftfile": "$(driftfile)", "servers": servers }' ),
template_data
is assigned a data container that is in this case constructed by mergedata()
so that only the necessary data is provided to the template. If template_data
is not explicitly provided, CFEngine uses datastate()
by default. It is considered best practice to provide explicit data as this makes it easier to delegate responsibility of the template and that data to different entities where neither are required to know anything about CFEngine itself and it's much more efficient to send the templating engine only the data the template actually uses.
Note, mergedata()
tries to expand bare values from CFEngine variables, so servers
will expand to the entire list of servers. The result of mergedata()
in the example is equivalent to this json:
{
"driftfile": "/var/lib/ntp/drift",
"servers": [ "time.nist.gov" ]
}
Now that we have dissected the policy, let's go ahead and give it a whirl.
Modify and run the policy
cf-agent -KIf update.cf;
info: Copied file '/var/cfengine/masterfiles/services/ntp.cf' to '/var/cfengine/inputs/services/ntp.cf.cfnew' (mode '600')
cf-agent -KI
info: Updated rendering of '/etc/ntp.conf' from mustache template 'inline'
info: files promise '/etc/ntp.conf' repaired
info: Executing 'no timeout' ... '/etc/init.d/ntpd restart'
info: Completed execution of '/etc/init.d/ntpd restart'
R: NTP service restarted after configuration change
More interestingly, if you examine the configuration file /etc/ntp.conf
, you will notice that it has been updated with the time server
(s) and driftfile
you had specified in the policy, for that specific operating system environment. This is the configuration that the NTP service has been restarted with.
grep -P "^(driftfile|server)" /etc/ntp.conf
driftfile /var/lib/ntp/drift
server time.nist.gov iburst
Mission Accomplished!
Instrumenting for tunability via augments
Next we will augment file/template management with data sourced from a JSON data file. This is a simple extension of what we have done previously illustrating how tunables in policy can be exposed and leveraged from a data feed.
CFEngine offers out-of-the-box support for reading and writing JSON data structures. In this tutorial, we will default the NTP configuration properties in policy, but provide a path for the properties to be overridden from Augments.
bundle agent ntp
{
vars:
linux::
"ntp_package_name" string => "ntp";
"config_file" string => "/etc/ntp.conf";
# Set the default value for driftfile
"driftfile"
string => "/var/lib/ntp/drift";
# Overwrite driftfile with value defined from Augments if it's provided
"driftfile"
string => "$(def.ntp[config][driftfile])",
if => isvariable( "def.ntp[config][driftfile]" );
# Set the default value for servers
"servers"
slist => { "time.nist.gov" };
# Overwrite servers with value defined from Augments if it's provided
"servers"
slist => getvalues( "def.ntp[config][servers]" ),
if => isvariable( "def.ntp[config][servers]" );
# For brevity, and since the template is small, we define it in-line
"config_template_string"
string => "# NTP Config managed by CFEngine
driftfile {{{driftfile}}}
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
{{#servers}}
server {{{.}}} iburst
{{/servers}}
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
";
redhat::
"ntp_service_name" string => "ntpd";
debian::
"ntp_service_name" string => "ntp";
packages:
"$(ntp_package_name)" -> { "StandardsDoc 3.2.1" }
policy => "present",
handle => "ntp_packages_$(ntp_package_name)",
classes => results("bundle", "ntp_package");
files:
"$(config_file)"
create => "true",
handle => "ntp_files_conf",
perms => mog( "644", "root", "root" ),
template_method => "inline_mustache",
edit_template_string => "$(config_template_string)",
template_data => mergedata( '{ "driftfile": "$(driftfile)", "servers": servers }' ),
classes => results( "bundle", "ntp_config" );
services:
"$(ntp_service_name)" -> { "StandardsDoc 3.2.2" }
service_policy => "start",
classes => results( "bundle", "ntp_service_running" );
ntp_config_repaired::
"$(ntp_service_name)" -> { "StandardsDoc 3.2.2" }
service_policy => "restart",
classes => results( "bundle", "ntp_service_config_change" );
reports:
ntp_service_running_repaired.inform_mode::
"NTP service started";
ntp_service_config_change_repaired.inform_mode::
"NTP service restarted after configuration change";
}
What does this policy do?
Let's review the changes to the vars promises as they were the only changes made.
vars
bundle agent ntp
{
vars:
linux::
"ntp_package_name" string => "ntp";
"config_file" string => "/etc/ntp.conf";
# Set the default value for driftfile
"driftfile"
string => "/var/lib/ntp/drift";
# Overwrite driftfile with value defined from Augments if it's provided
"driftfile"
string => "$(def.ntp[config][driftfile])",
if => isvariable( "def.ntp[config][driftfile]" );
# Set the default value for servers
"servers"
slist => { "time.nist.gov" };
# Overwrite servers with value defined from Augments if it's provided
"servers"
slist => getvalues( "def.ntp[config][servers]" ),
if => isvariable( "def.ntp[config][servers]" );
Notice two promises were introduced, one setting driftfile
to the value of $(def.ntp[config][driftfile])
if it is defined and one setting servers to the list of values for def.ntp[config][servers]
if it is defined. Augments allows for variables to be set in the def bundle scope very early before policy is evaluated.
Modify and run the policy
First modify services/ntp.cf
as shown previously (don't forget to check syntax with cf-promises
after modification), then run the policy.
cf-agent -KIf update.cf
info: Copied file '/var/cfengine/masterfiles/services/ntp.cf' to '/var/cfengine/inputs/services/ntp.cf.cfnew' (mode '600')
info: Copied file '/var/cfengine/masterfiles/def.json' to '/var/cfengine/inputs/def.json.cfnew' (mode '600')
cf-agent -KI
We do not expect to see the ntp configuration file modified or the service to be restarted since we have only instrumented the policy so far.
Now, let's modify def.json
(in the root of masterfiles) and define some different values for driftfile
and servers
.
Modify def.json
so that it looks like this:
{
"inputs": [ "services/ntp.cf" ],
"vars": {
"control_common_bundlesequence_end": [ "ntp" ],
"ntp": {
"config": {
"driftfile": "/tmp/drift",
"servers": [ "0.north-america.pool.ntp.org", "1.north-america.pool.ntp.org",
"2.north-america.pool.ntp.org", "3.north-america.pool.ntp.org" ]
}
}
}
}
Now, let's validate the JSON and force a policy run and inspect the result.
python -m json.tool < def.json
{
"inputs": [
"services/ntp.cf"
],
"vars": {
"control_common_bundlesequence_end": [
"ntp"
],
"ntp": {
"config": {
"driftfile": "/tmp/drift",
"servers": [
"0.north-america.pool.ntp.org",
"1.north-america.pool.ntp.org",
"2.north-america.pool.ntp.org",
"3.north-america.pool.ntp.org"
]
}
}
}
}
cf-agent -KI
info: Updated rendering of '/etc/ntp.conf' from mustache template 'inline'
info: files promise '/etc/ntp.conf' repaired
info: Executing 'no timeout' ... '/etc/init.d/ntpd restart'
info: Completed execution of '/etc/init.d/ntpd restart'
R: NTP service restarted after configuration change
info: Can not acquire lock for 'ntp' package promise. Skipping promise evaluation
info: Can not acquire lock for 'ntp' package promise. Skipping promise evaluation
grep -P "^(driftfile|server)" /etc/ntp.conf
driftfile /tmp/drift
server 0.north-america.pool.ntp.org iburst
server 1.north-america.pool.ntp.org iburst
server 2.north-america.pool.ntp.org iburst
server 3.north-america.pool.ntp.org iburst
Mission Accomplished!
You have successfully completed this tutorial that showed you how to write a simple policy to ensure that NTP is installed, running and configured appropriately.
Package management
Package management is a critical task for any system administrator. In this tutorial we will show you how easy it is to install, manage and remove packages using CFEngine.
As a first example, we will use CFEngine to update OpenSSL, which is timely given the recent disclosure of the Heartbleed vulnerability. If we simply want to make sure the latest version of OpenSSL is installed in all our hosts, we can use the packages promise type, like this:
body common control
{
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent manage_packages
{
packages:
"openssl"
policy => "present",
version => "latest",
package_module => yum;
}
bundle agent __main__
{
methods:
"manage_packages";
}
The package_module
promise attribute tells CFEngine which package manager we
want to use. Defaults can be set up by using the package_module
common control
attribute. When we run this on an CentOS 6 system, we can verify the openssl
version before and after running the policy, and we get the following output:
yum list installed | grep openssl
openssl.x86_64 1.0.0-27.el6 @anaconda-CentOS-201303020151.x86_64/6.4
openssl-devel.x86_64 1.0.0-27.el6 @anaconda-CentOS-201303020151.x86_64/6.4
cf-agent -K ./manage_packages.cf
yum list installed | grep openssl
openssl.x86_64 1.0.1e-42.el6 @base
openssl-devel.x86_64 1.0.1e-42.el6 @base
Additionally, you may want to make sure certain packages are not installed on the system. On my CentOS 6 system, I can see that the telnet package is installed.
yum list installed | grep telnet
telnet.x86_64 1:0.17-48.el6 @base
which telnet
/usr/bin/telnet
Making sure this package is removed from the system is easy. Let's add one more promise to our previous policy, this time using the absent policy:
body common control
{
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent manage_packages
{
packages:
"openssl"
policy => "present",
version => "latest",
package_module => yum;
"telnet"
policy => "absent",
package_module => yum;
}
bundle agent __main__
{
methods:
"manage_packages";
}
Note that we leave the previous line in place. This way, CFEngine will continue to ensure that the openssl package is always updated to its latest version. We can now see the policy in action:
# cf-agent -K ./manage_packages.cf
# yum list installed | grep telnet
# which telnet
/usr/bin/which: no telnet in (/sbin:/bin:/usr/sbin:/usr/bin:/var/cfengine/bin)
The packages promise also supports version pinning, so that you can specify exactly the version you want to have installed. It is modular and extensible, so that it is easy to add support for new platforms and package managers. For complete documentation, please have a look at the reference manual for the packages promise.
Of course, running the policy by hand is only good for initial testing. Once your policy works the way you need, you will want to deploy it to your entire infrastructure by integrating your policy into your regular cf-agent execution, thus making sure that the desired state of your packages is always kept automatically on all your machines. To do so, you need to do the following:
Copy manage_packages.cf
to /var/cfengine/masterfiles/
on your policy hub. In
/var/cfengine/masterfiles/def.json
, add manage_packages.cf
to the inputs
declaration, and manage_packages
to the bundlesequence declaration.
{
"inputs": [ "manage_packages.cf" ],
"vars": {
"control_common_update_bundlesequence_end": [ "manage_packages" ]
}
}
Run cf-promises
on the policy to verify that there are no errors.
cf-promises -cf /var/cfengine/masterfiles/promises.cf
Wait a few minutes for the new policy to propagate and start taking effect in your entire infrastructure. This is where the real power of CFEngine becomes apparent! With few changes in a single place, you can control the desired state of your entire infrastructure, whether it's composed of a few or many thousands of machines.
Managing processes and services
Ensuring a particular process is running on a system is a common task for a system administrator, as processes are what provide all services available on a computer system.
Using CFEngine to ensure certain processes are running is extremely easy.
- Create the policy
Create a new file called ensure_process.cf
:
body file control
{
inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent main
{
processes:
"/usr/sbin/ntpd"
restart_class => "ntpd_not_running";
commands:
ntpd_not_running::
"/etc/init.d/ntp start";
}
This example is designed to be run on an Ubuntu 12.04 system, and assumes the ntp package is already installed.
Let us quickly explain this code:
The body file control
construct, which instructs CFEngine to load the CFEngine
standard library.
The processes:
tells cf-agent that the promises are related to
processes. Then a promise checks for the existence of a running process whose
name matches the string /usr/sbin/ntpd
. If the process is found, nothing
happens. But if it is not found, the ntpd_not_running
class (a class is a named
boolean attribute in the CFEngine policy language which can be used for decision
making) will be defined.
Finally, the commands:
line tells cf-agent that the following promises are
related to executing commands. The ntpd_not_running::
line restricts the context
to so that the following commands will only be run if the expression evaluates
to true.
- Testing the policy
First, we verify that the ntpd process is not running:
ps axuww | grep ntp
Then we run our CFEngine policy:
cf-agent -f ./ensure_process.cf
2014-03-20T06:33:56+0000 notice: /default/main/commands/'/etc/init.d/ntp start'[0]: Q: "...init.d/ntp star": * Starting NTP server ntpd
Q: "...init.d/ntp star": ...done.
Finally, we verify that ntpd is now running on the system:
ps axuww | grep ntp
ntp 5756 0.3 0.1 37696 2172 ? Ss 06:33 0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 104:110
Congratulations!
That's it! Every time CFEngine runs the policy, it will check for the process, and if it's not there, will start it. This is how CFEngine maintains your system in the correct, desired state.
Writing CFEngine policy
To define new Desired States in CFEngine, you need to write policy files. These are plain text-files, traditionally with a .cf
extension.
/var/cfengine/inputs and promises.cf
In CFEngine, cf-agent
executes all policies. cf-agent
runs every 5 minutes
by default, and it executes policies found locally in the /var/cfengine/inputs
directory. The default policy entry is a file called promises.cf
. In this file
you normally reference bundles and other policy files.
Bundles, promise types, and classes oh my!
These concepts are core to CFEngine so they are covered in brief here. For more detailed information see the Language concepts section of the Reference manual.
Bundles
Bundles are re-usable and blocks of CFEngine policy. The following defines a bundle called my_test
, and it is a bundle for the agent.
bundle agent my_test
{
# ...
}
A bundle contains one or more promise types.
Promise types
Think of a promise type as a way to abstract yourself away from details. For instance, there is a promise type called users. This promise type allows you to manage local users on your system. Instead of using low-level commands to manage users, with the promise type, you only have one simple syntax that works across all operating systems.
The most frequently used promise types are vars
, classes
, files
,
packages
, users
, services
, commands
and reports
. Whenever you want to
do some file configuration for example, you would use the files promise type.
The following policy ensures the existence of the /tmp/hello-world
file:
files:
"/tmp/hello-world"
create => "true";
When defining desired states it is important to be clear about when and where you want this policy to apply. For that, CFEngine has the concept of classes.
Classes
A class is an identifier which is used by the agent to decide when and where a part of a policy shall run. A class can either be user-defined, a so called soft-class, or it can be a hard class which is automatically discovered and defined by cf-agent during each run. Popular classes include any which means any or all hosts, policy_server which means the host is a policy server. There are more than 50 hard classes, and combined with regular expressions this gives you very granular control.
To see a list of available classes on your host, just type the following command:
cf-promises --show-classes
Running policy
Now let's put the bundle, promise type and class components together in a
final policy. As for classes we will use linux to define that the file
/tmp/hello-world
must exists on all hosts of type linux:
bundle agent my_test
{
files:
linux::
"/tmp/hello-world"
create => "true";
}
bundle agent __main__
{
methods: "my_test";
}
Let's save this policy in /tmp/my-policy.cf
.
You can now run this policy either in Distributed (client-server) System or in a Stand Alone system. The next two sections will cover each of the options.
Option#1: Running the policy on a stand alone system
Since CFEngine is fully distributed we can run policies locally. This can come in handy as the result of a run is instant, especially during the design phase where you would like to test out various policies.
To run the file locally, you can log into any of your hosts that has CFEngine installed and follow these steps. For this tutorial, use your Policy Server for this as it is the same cf-agent that runs on the hosts as on the Policy Server.
Tip: Whenever you make or modify a policy, you can use the cf-promises
command to run a syntax check:
cf-promises -f /tmp/my-policy.cf
Unless you get any output, the syntax is correct. Now, to run this policy, simply type:
cf-agent -Kf /tmp/my-policy.cf
As you can see, the response is immediate! Running CFEngine locally like this is ideal for testing out new policies. To check that the file has been successfully created type:
ls /tmp/hello-world -l
If you want to see what the agent is doing during its run, you can run the agent in verbose mode. Try:
cf-agent -Kf /tmp/my-policy.cf --verbose
In a Stand Alone system, to make and run a policy remember to:
Option#2: Running the Policy on a Distributed System
CFEngine is designed for large-scale systems. It is fully distributed which means that all the logic and decision making takes place on the end-points, or hosts as we call them in CFEngine. The hosts fetch their policies from one central distribution point. To continue with this option you need to have CFEngine running on at least one host and one policy server.
The CFEngine Server typically acts as a policy distribution point. On each host, the cf-agent process runs regularly. This process will by default, every 5 minutes, try to connect to cf-serverd on the policy server to check for policy updates.
By default cf-serverd
will serve policy from the /var/cfengine/masterfiles
directory. When the content changes, cf-agent will download the updated files to
/var/cfengine/inputs
before executing them locally.
This means that by default you should store all your policies in the
/var/cfengine/masterfiles
directory on your policy server. So, now create
/var/cfengine/masterfiles/my-policy.cf
with the content of the test policy
previously authored.
NOTE: We recommend that you use a version control system to store the audit log of your policy.
Now we need to tell CFEngine that there is a new policy in town:
- Create
/var/cfengine/masterfiles/def.json
with the following content:
{
"inputs": [ "my-policy.cf" ]
}
On the policy server you can run the following command to make sure the syntax is correct.
cf-agent -cf /var/cfengine/masterfiles/promises.cf
After some period of time (CFEngine runs by default every 5 minutes), log in to
any of the bootstrapped clients and you will find the /tmp/-hello-world
file
there.
Whenever a host connects to the Policy Server, the host will ensure that it has
the my-policy.cf
file from the masterfiles directory exists in the local
inputs directory. def.json
will also be downloaded with the new
instruction that there is a new policy. Within 5 minutes, whether you have 5
Linux hosts or 50,000 Linux hosts, they will all have the /tmp/hello-world
file
on their system. Yeah!
If you delete the file, it will be restored by CFEngine at its next run. We call this a promise repaired. If the file exists during a run, the result would be promise kept.
Congratulations! You now have the basic knowledge needed to write and run CFEngine policies. Let's continue with an example on how to manage users. Click here to continue.
Distributing files from a central location
CFEngine can manage many machines simply by distributing policies to all its hosts. This tutorial describes how to distribute files to hosts from a central policy server location. For this example, we will distribute software patches.
Files are centrally stored on the policy server (hub). In our example, they are stored in /storage/patches
.
These patch files must also exist on the agent host (client) in /storage/deploy/patches
. To do this,
perform the following instructions:
Check out masterfiles from your central repository
CFEngine stores the master copy of all policy in the /var/cfengine/masterfiles
directory.
Ensure that you are working with the latest version of your masterfiles
.
git clone url
or
git pull origin master
Make policy changes
Define locations
Before files can be copied we must know where files should be copied from and where files should be copied to. If these locations are used by multiple components, then defining them in a common bundle can reduce repetition. Variables and classes that are defined in common bundles are accessible by all CFEngine components. This is especially useful in the case of file copies because the same variable definition can be used both by the policy server when granting access and by the agent host when performing the copy.
The policy framework includes a common bundle called def
. In this example, we
will add two variables--dir_patch_store
and dir_patch_deploy
--to this existing bundle.
These variables provide path definitions for storing and deploying patches.
Add the following variable information to the masterfiles/def.cf
file:
"dir_patch_store"
string => "/storage/patches",
comment => "Define patch files source location",
handle => "common_def_vars_dir_patch_store";
"dir_patch_deploy"
string => "/storage/deploy/patches",
comment => "Define patch files deploy location",
handle => "common_def_vars_dir_patch_deploy";
}
These common variables can be referenced from the rest of the policy by using their fully
qualified names,
$(def.dir_patch_store)
and $(def.dir_patch_deploy)
Grant file access
Access must be granted before files can be copied. The right to access a file
is provided by cf-serverd
, the server component of CFEngine. Enter access information using the access
promise type in a server
bundle. The default access rules defined by the MPF (Masterfiles Policy Framework) can be found in
controls/cf_serverd.cf
.
There is no need to modify the vendored policy, instead define your own server bundle. For our example, add the following to services/main.cf
:
bundle server my_access_rules
{
access:
"$(def.dir_patch_store)"
handle => "server_access_grant_locations_files_patch_store_for_hosts",
admit => { ".*$(def.domain)", @(def.acl) },
comment => "Hosts need to download patch files from the central location";
}
Create a custom library for reusable synchronization policy
You might need to frequently synchronize or copy a directory structure from the policy server to an agent host. Thus, identifying reusable parts of policy and abstracting them for later use is a good idea. This information is stored in a custom library.
Create a custom library called lib/custom/files.cf
. Add the following content:
bundle agent sync_from_policyserver(source_path, dest_path)
# @brief Sync files from the policy server to the agent
#
# @param source_path Location on policy server to copy files from
# @param dest_path Location on agent host to copy files to
{
files:
"$(dest_path)/."
handle => "sync_from_policy_server_files_dest_path_copy_from_source_path_sys_policy_hub",
copy_from => sync_cp("$(source_path)", "$(sys.policy_hub)"),
depth_search => recurse("inf"),
comment => "Ensure files from $(sys.policy_hub):$(source_path) exist in $(dest_path)";
}
This reusable policy will be used to synchronize a directory on the policy server to a directory on the agent host.
Create a patch policy
Organize in a way that makes the most sense to you and your team. We recommend organizing policy by services.
Create services/patching.cf
with the following content:
# Patching Policy
bundle agent patching
# @brief Ensure various aspects of patching are handeled
# We can break down the various parts of patching into separate bundles. This
# allows us to become less overwhelmed by details if numerous specifics
# exist in one or more aspect for different host classifications.
{
methods:
"Patch Distribution"
handle => "patching_methods_patch_distribution",
usebundle => "patch_distribution",
comment => "Ensure patches are properly distributed";
}
bundle agent patch_distribution
# @brief Ensures that our patches are distributed to the proper locations
{
files:
"$(def.dir_patch_deploy)/."
handle => "patch_distribution_files_def_dir_patch_deploy_exists",
create => "true",
comment => "If the destination directory does not exist, we have no place
to which to copy the patches.";
methods:
"Patches"
handle => "patch_distribution_methods_patches_from_policyserver_def_dir_patch_store_to_def_dir_patch_deploy",
usebundle => sync_from_policyserver("$(def.dir_patch_store)", "$(def.dir_patch_deploy)"),
comment => "Patches need to be present on host systems so that we can use
them. By convention we use the policy server as the central
distribution point.";
}
The above policy contains two bundles. We have separated a top-level patching
bundle from a more specific patch_distribution
bundle. This is an
illustration of how to use bundles in order to abstract details. You
might, for example, have some hosts that you don't want to fully
synchronize so you might use a different method or copy from a
different path. Creating numerous bundles allows you to move those details away from the top
level of what is involved in patching. If people are interested in what
is involved in patch distribution, they can view that bundle for specifics.
Integrate the policy
Now that all the pieces of the policy are in place, they must be integrated
into the policy so they can be activated. Add each policy file to the inputs
section which is found under body common control
. Once the policy file is included in
inputs, the bundle can be activated. Bundles can be activated by adding them to either the
bundlesequence
or they can be called as a methods
type promise.
Add the following entries to promises.cf
under body common control
-> inputs
:
"lib/custom/files.cf",
"services/patching.cf",
and the following to promises.cf
under body common control
-> bundlesequence
:
"patching",
Now that all of the policy has been edited and is in place, check for syntax errors by
running cf-promises -f ./promises.cf
. This promise is activated from the service_catalogue
bundle.
Commit Changes
Set up trackers in the Mission Portal (Enterprise Users Only)
Before committing the changes to your repository, log in to the Mission Portal and set up a Tracker so that you can see the policy as it goes out. To do this, perform the following:
Navigate to the Hosts section. Select All hosts. Select the Events tab, located in the right-hand panel. Click Add new tracker.
Name it Patch Failure. Set the Report Type to Promise not Kept. Under Watch, enter .patch. Set the Start Time to Now and then click Done to close the Start Time window. Click Start to save the new tracker. This tracker watches for any promise handle that includes the string patch where a promise is not kept.
Add another tracker called Patch Repaired. Set the Report Type to Promise Repaired. Enter the same values as above for Watch and Start Time. Click Start to save the new tracker. This tracker allows you to see how the policy reacts as it is activated on your infrastructure.
Deploy changes (Enterprise and Community Users)
Always inspect what you expect. git status
shows the status of your current branch.
git status
Inspect the changes contained in each file.
git diff file
Once satisfied, add them to Git's commit staging area.
git add file
Iterate over using git diff, add, and status until all of the changes that you expected are listed as Changes to be committed. Check the status once more before you commit the changes.
git status
Commit the changes to your local repository.
git commit
Push the changes to the central repository so they can be pulled down to your policy server for distribution.
git push origin master
File editing
Prerequisites
- Read the tutorial Tutorial for running examples
- Ensure you have read and understand the section on how to make an example stand alone
- Ensure you have read the note at the end of that section regarding modification of the body common control to the following:
body common control
{
inputs => {
"libraries/cfengine_stdlib.cf",
};
}
Note: This change is not necessary for supporting each of the examples in this tutorial. It will be included only in those examples that require it.
List Files
Note: The following workflow assumes the directory /home/user already exists. If it does not either create the directory or adjust the example to a path of your choosing.
Create a file /var/cfengine/masterfiles/file_test.cf that includes the following text:
file_test.cfbundle agent list_file { vars: "ls" slist => lsdir("/home/user", "test_plain.txt", "true"); reports: "ls: $(ls)"; }
Run the following command to remove any existing test file at the location we wish to use for testing this example:
commandrm /home/user/test_plain.txt
Test to ensure there is no file /home/user/test_plain.txt, using the following command (the expected result is that there should be no file listed at the location /home/user/test_plain.txt):
commandls /home/user/test_plain.txt
Run the following command to instruct CFEngine to see if the file exists (the expected result is that no report will be generated (because the file does not exist):
command/var/cfengine/bin/cf-agent --no-lock --file /var/cfengine/masterfiles/file_test.cf --bundlesequence list_file
Create a file for testing the example, using the following command:
commandtouch /home/user/test_plain.txt
Run the following command to instruct CFEngine to search for the file (the expected result is that a report will be generated, because the file exists):
command/var/cfengine/bin/cf-agent --no-lock --file /var/cfengine/masterfiles/file_test.cf --bundlesequence list_file
Double check the file exists, using the following command (the expected result is that there will be a file listed at the location /home/user/test_plain.txt):
commandls /home/user/test_plain.txt
Run the following command to remove the file:
commandrm /home/user/test_plain.txt
Create a File
bundle agent testbundle
{
files:
"/home/user/test_plain.txt"
perms => system,
create => "true";
}
bundle agent list_file
{
vars:
"ls"
slist => lsdir("/home/user", "test_plain.txt", "true");
reports:
"ls: $(ls)";
}
bundle agent list_file_2
{
vars:
"ls"
slist => lsdir("/home/user", "test_plain.txt", "true"); reports:
"ls: $(ls)";
}
body perms system
{
mode => "0640";
}
ls /home/user/test_plain.txt
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence list_file,testbundle,list_file_2
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence list_file,list_file_2
ls /home/user/test_plain.txt
rm /home/user/test_plain.txt
Delete a File
body common control
{
inputs => {
"libraries/cfengine_stdlib.cf",
};
}
bundle agent testbundle
{
files:
"/home/user/test_plain.txt"
perms => system,
create => "true";
}
bundle agent test_delete
{
files:
"/home/user/test_plain.txt"
delete => tidy;
}
bundle agent list_file
{
vars:
"ls"
slist => lsdir("/home/user", "test_plain.txt", "true");
reports:
"ls: $(ls)";
}
bundle agent list_file_2
{
vars:
"ls"
slist => lsdir("/home/user", "test_plain.txt", "true");
reports:
"ls: $(ls)";
}
body perms system
{
mode => "0640";
}
rm /home/user/test_plain.txt
ls /home/user/test_plain.txt
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence list_file,testbundle,list_file_2
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence list_file,list_file_2
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence list_file,test_delete,list_file_2
ls /home/user/test_plain.txt
rm /home/user/test_plain.txt
(last command will throw an error because the file doesn't exist!)
Modify a File
rm /home/user/test_plain.txt
ls /home/user/test_plain.txt
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence list_file,testbundle,list_file_2
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence list_file,list_file_2
body common control
{
inputs => {
"libraries/cfengine_stdlib.cf",
};
}
bundle agent testbundle
{
files:
"/home/user/test_plain.txt"
perms => system,
create => "true";
}
bundle agent test_delete
{
files:
"/home/user/test_plain.txt"
delete => tidy;
}
bundle agent list_file
{
vars:
"ls"
slist => lsdir("/home/user", "test_plain.txt", "true");
reports:
"ls: $(ls)";
}
bundle agent list_file_2
{
vars:
"ls"
slist => lsdir("/home/user", "test_plain.txt", "true");
reports:
"ls: $(ls)";
}
# Finds the file, if exists calls bundle to edit line
bundle agent outer_bundle_1
{
files:
"/home/user/test_plain.txt"
create => "false",
edit_line => inner_bundle_1;
}
# Finds the file, if exists calls bundle to edit line
bundle agent outer_bundle_2
{
files:
"/home/user/test_plain.txt"
create => "false",
edit_line => inner_bundle_2;
}
# Inserts lines
bundle edit_line inner_bundle_1
{
vars:
"msg"
string => "Helloz to World!";
insert_lines:
"$(msg)";
}
# Replaces lines
bundle edit_line inner_bundle_2
{
replace_patterns:
"Helloz to World!"
replace_with => hello_world;
}
body replace_with hello_world
{
replace_value => "Hello World";
occurrences => "all";
}
body perms system
{
mode => "0640";
}
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence list_file,test_delete,list_file_2
ls /home/user/test_plain.txt
rm /home/user/test_plain.txt
Copy a file and edit its text
body common control
{
inputs => {
"libraries/cfengine_stdlib.cf",
};
}
bundle agent testbundle
{
files:
"/home/ichien/test_plain.txt"
perms => system,
create => "true";
reports:
"test_plain.txt has been created";
}
bundle agent test_delete
{
files:
"/home/ichien/test_plain.txt"
delete => tidy;
}
bundle agent do_files_exist
{
vars:
"mylist"
slist => {
"/home/ichien/test_plain.txt",
"/home/ichien/test_plain_2.txt",
};
classes:
"exists"
expression => filesexist("@(mylist)");
reports:
exists::
"test_plain.txt and test_plain_2.txt files exist";
!exists::
"test_plain.txt and test_plain_2.txt files do not exist";
}
bundle agent do_files_exist_2
{
vars:
"mylist"
slist => {
"/home/ichien/test_plain.txt",
"/home/ichien/test_plain_2.txt"
};
classes:
"exists"
expression => filesexist("@(mylist)");
reports:
exists::
"test_plain.txt and test_plain_2.txt files both exist";
!exists::
"test_plain.txt and test_plain_2.txt files do not exist";
}
bundle agent list_file_1
{
vars:
"ls1"
slist => lsdir("/home/ichien", "test_plain.txt", "true");
"ls2"
slist => lsdir("/home/ichien", "test_plain_2.txt", "true");
"file_content_1"
string => readfile("/home/ichien/test_plain.txt", "33");
"file_content_2"
string => readfile("/home/ichien/test_plain_2.txt", "33");
reports:
# "ls1: $(ls1)";
# "ls2: $(ls2)";
"Contents of /home/ichien/test_plain.txt = $(file_content_1)";
"Contents of /home/ichien/test_plain_2.txt = $(file_content_2)";
}
bundle agent list_file_2
{
vars:
"ls1"
slist => lsdir("/home/ichien", "test_plain.txt", "true");
"ls2"
slist => lsdir("/home/ichien", "test_plain_2.txt", "true");
"file_content_1"
string => readfile("/home/ichien/test_plain.txt", "33");
"file_content_2"
string => readfile("/home/ichien/test_plain_2.txt", "33");
reports:
# "ls1: $(ls1)";
# "ls2: $(ls2)";
"Contents of /home/ichien/test_plain.txt = $(file_content_1)";
"Contents of /home/ichien/test_plain_2.txt = $(file_content_2)";
}
bundle agent outer_bundle_1
{
files:
"/home/ichien/test_plain.txt"
create => "false",
edit_line => inner_bundle_1;
}
# Copies file
bundle agent copy_a_file
{
files:
"/home/ichien/test_plain_2.txt"
copy_from => local_cp("/home/ichien/test_plain.txt");
reports:
"test_plain.txt has been copied to test_plain_2.txt";
}
bundle agent outer_bundle_2
{
files:
"/home/ichien/test_plain_2.txt"
create => "false",
edit_line => inner_bundle_2;
}
bundle edit_line inner_bundle_1
{
vars:
"msg"
string => "Helloz to World!";
insert_lines:
"$(msg)";
reports:
"inserted $(msg) into test_plain.txt";
}
bundle edit_line inner_bundle_2
{
replace_patterns:
"Helloz to World!"
replace_with => hello_world;
reports:
"Text in test_plain_2.txt has been replaced";
}
body replace_with hello_world
{
replace_value => "Hello World";
occurrences => "all";
}
body perms system
{
mode => "0640";
}
/var/cfengine/bin/cf-agent --no-lock --file ./file_test.cf --bundlesequence test_delete,do_files_exist,testbundle,outer_bundle_1,copy_a_file,do_files_exist_2,list_file_1,outer_bundle_2,list_file_2
Reporting and remediation of security vulnerabilities
Prerequisites
- CFEngine 3.6 Enterprise Hub
- At least one client vulnerable to CVE-2014-6271
Overview
Remediating security vulnerabilities is a common issue. Sometimes you want to know the extent to which your estate is affected by a threat. Identification of affected systems can help you prioritize and plan remediation efforts. In this tutorial you will learn how to inventory your estate and build alerts to find hosts that are affected by the #shellshock exploit. After identifying the affected hosts you will patch a subset of the hosts and then be able to see the impact on your estate. The same methodology can be applied to other issues.
Note: The included policy does not require CFEngine Enterprise. Only the reporting functionality (Mission Portal) requires the Enterprise version.
Inventory CVE-2013-6271
Writing inventory policy with CFEngine is just like any other CFEngine policy,
except for the addition of special meta
attributes used to augment the
inventory interface. First you must know how to collect the information you
want. In this case we know that a vulnerable system will have the word
vulnerable listed in the output of the command
env x='() { :;}; echo vulnerable' $(bash) -c 'echo testing CVE-2014-6271'
.
This bundle will check if the host is vulnerable to the CVE, define a class CVE_2014_6217 if it is vulnerable and augment Mission Portals Inventory interface in CFEngine Enterprise.
bundle agent inventory_CVE_2014_6271
{
meta:
"description" string => "Remote exploit vulnerability in bash http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-6271";
"tags" slist => { "autorun" };
vars:
"env" string => "$(paths.env)";
"bash" string => "/bin/bash";
"echo" string => "$(paths.echo)";
"test_result" string => execresult("$(env) x='() { :;}; $(echo) vulnerable' $(bash) -c 'echo testing CVE-2014-6271'", "useshell");
CVE_2014_6271::
"vulnerable"
string => "CVE-2014-6271",
meta => { "inventory", "attribute_name=Vulnerable CVE(s)" };
classes:
"CVE_2014_6271"
expression => regcmp( "vulnerable.*", "$(test_result)" ),
scope => "namespace",
persistence => "10",
comment => "We persist the class for 2 agent runs so that bundles
activated before this bundle can use the class on the next
agent execution to coordinate things like package updates.";
reports:
DEBUG|DEBUG_cve_2014_6217::
"Test Result: $(test_result)";
CVE_2014_6271.(inform_mode|verbose_mode)::
"Tested Vulnerable for CVE-2014-6271: $($(this.bundle)_meta.description)";
}
What does this inventory policy do?
Meta type promises are used to attach additional information to bundles. We have set 'description' so that future readers of the policy will know what the policy is for and how to get more information on the vulnerability. For the sake of simplicity in this example set 'autorun' as a tag to the bundle. This makes the bundle available for automatic activation when using the autorun feature in the Masterfiles Policy Framework.
Next we set the paths to the binaries that we will use to exeucte our test command. As of this writing the paths for 'env' and 'echo' are both in the standard libraries paths bundle, but 'bash' is not. Note that you may need to adjust the path to bash for your platforms. Then we run our test command and place the command output into the 'test_result' variable. Since we have no CVE_2014_6271 class defined yet, the next promise to set the variable 'vulnerable' to 'CVE-2014-6271' will be skipped on the first pass. Then the classes type promise is evaluated and defines the class CVE_2014_6271 if the output matches the regular expression 'vulnerable.*'. Finally the reports are evaluated before starting the second pass. If the class 'DEBUG' or 'DEBUG_inventory_CVE_2014_6271' is set the test command output will be shown, and if the vulnerability is present agent is running in inform or verbose mode message indicating the host is vulnerable along with the description will be output.
On the second pass only that variable 'vulnerable' will be set with the value 'CVE-2014-6271' if the host is vulnerable. Note how this variable tagged with 'inventory' and 'attribute_name='. These are special meta tags that CFEngine Enterprise uses in order to display information.
Deploy the policy
As noted previously, in this example we will use autorun for simplicity. Please
ensure that the class "services_autorun" is defined. The easiest way to do this
is to change "services_autorun" expression => "!any";
to "services_autorun"
expression => "any";
in def.cf
.
Once you have autorun enabled you need only save the policy into
services/autorun/inventory_CVE_2014_6271.cf
.
Report on affected system inventory
Within 20 minutes of deploying the policy you should be able to see results in the Inventory Reporting interface.
A new Inventory attribute 'Vulnerable CVE(s)' is available.
Report showing CVEs that each host is vulnerable to.
Chart the Vulnerable CVE(s) and get a visual breakdown.
Build Dashboard Widget with Alerts
Let's add alerts for CVE(s) to the dashboard.
Give the dashboard widget a name.
Configure an general CVE alert for the dashboard.
Add an additional alert for this specific CVE.
See the dashboard alert in action.
Remediate vulnerabilities
Now that we know the extent of exposure lets ensure bash gets updated on some
of the affected systems. Save the following policy into
services/autorun/remediate_CVE_2014_6271.cf
bundle agent remediate_CVE_2014_6271
{
meta:
"tags" slist => { "autorun" };
classes:
"allow_update" or => { "hub", "host001" };
methods:
allow_update.CVE_2014_6271::
"Upgrade_Bash"
usebundle => package_latest("bash");
}
What does this remediation policy do?
For simplicity of the example this policy defines the class allow_update on hub and host001, but you could use any class that makes sense to you. If the allow_update class is set, and the class CVE_2014_6271 is defined (indicating the host is vulnerable) then the policy ensures that bash is updated to the latest version available.
Report on affected systems inventory after remediation
Within 20 minutes or so of the policy being deployed you will be able to report on the state of remediation.
See the remediation efforts relfected in the dashboard.
Drill down into the dashboard and alert details.
Run an Inventory report to see hosts and their CVE status.
Chart the Vulnerable CVE(s) and get a visual breakdown.
Summary
In this tutorial you have learned how to use the reporting and inventory features of CFEngine Enterprise to discover and report on affected systems before and after remediation efforts.
Masterfiles Policy Framework upgrade
Upgrading the Masterfiles Policy Framework (MPF) is the first step in upgrading CFEngine from one version to another. The MPF should always be the same version or newer than the binary versions running.
Upgrading the MPF is not an exact process as the details highly depend on the specifics of the changes made to the default policy. This example leverages git
and shows an example of upgrading a simple policy set based on 3.18.0
to 3.21.2
and can be used as a reference for upgrading your own policy sets.
Prepare a Git clone of your working masterfiles
We will perform the integration work in /tmp/MPF-upgrade/integration
. masterfiles
should exist in the integration directory and is expected to be both the root of your policy set and a git
repository.
Validating expectations
From /tmp/MPF-upgrade/integration/masterfiles
. Let's inspect what we expect.
Is it the root of a policy set? promises.cf
will be present if so.
export INTEGRATION_ROOT="/tmp/MPF-upgrade/integration"
cd $INTEGRATION_ROOT/masterfiles
if [ -e "promises.cf" ]; then
echo "promise.cf exists, it's likely the root of a policy set"
else
echo "promises.cf is missing, $INTEGRATION_ROOT/masterfiles does not seem like the root of a policy set"
fi
promise.cf exists, it's likely the root of a policy set
Let's see what version of the MPF we are starting from by looking at version
in body common control
of promises.cf
.
grep -P "\s+version\s+=>" $INTEGRATION_ROOT/masterfiles/promises.cf 2>&1 \
|| echo "promises.cf is missing, $INTEGRATION_ROOT/masterfiles does not seem to be the root of a policy set"
version => "CFEngine Promises.cf 3.18.0";
And finally, is it a git repository, what is the last commit?
git status \
|| echo "$INTEGRATION_ROOT/masterfiles does not appear to be a git repository!" \
&& git log -1
On branch master
nothing to commit, working tree clean
commit f4c0e120b0b45bcb9ede01ed8fb465f40b4b1e6f
Author: Nick Anderson <nick@cmdln.org>
Date: Wed Jul 26 18:43:06 2023 -0500
CFEngine Policy set prior to upgrade
Merge upstream changes from the MPF into your policy
Remove everything except the .git
directory
By first removing everything we will easily be able so see which files are new, changed, moved or removed upstream.
rm -rf *
Check git status
to see that all the files have been deleted and are not staged for commit.
git status
On branch master
Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
deleted: cfe_internal/CFE_cfengine.cf
deleted: cfe_internal/core/deprecated/cfengine_processes.cf
deleted: cfe_internal/core/host_info_report.cf
deleted: cfe_internal/core/limit_robot_agents.cf
deleted: cfe_internal/core/log_rotation.cf
deleted: cfe_internal/core/main.cf
deleted: cfe_internal/core/watchdog/templates/watchdog-windows.ps1.mustache
deleted: cfe_internal/core/watchdog/templates/watchdog.mustache
deleted: cfe_internal/core/watchdog/watchdog.cf
deleted: cfe_internal/enterprise/CFE_hub_specific.cf
deleted: cfe_internal/enterprise/CFE_knowledge.cf
deleted: cfe_internal/enterprise/federation/federation.cf
deleted: cfe_internal/enterprise/file_change.cf
deleted: cfe_internal/enterprise/ha/ha.cf
deleted: cfe_internal/enterprise/ha/ha_def.cf
deleted: cfe_internal/enterprise/ha/ha_update.cf
deleted: cfe_internal/enterprise/main.cf
deleted: cfe_internal/enterprise/mission_portal.cf
deleted: cfe_internal/enterprise/templates/httpd.conf.mustache
deleted: cfe_internal/enterprise/templates/runalerts.php.mustache
deleted: cfe_internal/enterprise/templates/runalerts.sh.mustache
deleted: cfe_internal/recommendations.cf
deleted: cfe_internal/update/cfe_internal_dc_workflow.cf
deleted: cfe_internal/update/cfe_internal_update_from_repository.cf
deleted: cfe_internal/update/lib.cf
deleted: cfe_internal/update/systemd_units.cf
deleted: cfe_internal/update/update_bins.cf
deleted: cfe_internal/update/update_policy.cf
deleted: cfe_internal/update/update_processes.cf
deleted: cfe_internal/update/windows_unattended_upgrade.cf
deleted: controls/cf_agent.cf
deleted: controls/cf_execd.cf
deleted: controls/cf_hub.cf
deleted: controls/cf_monitord.cf
deleted: controls/cf_runagent.cf
deleted: controls/cf_serverd.cf
deleted: controls/def.cf
deleted: controls/def_inputs.cf
deleted: controls/reports.cf
deleted: controls/update_def.cf
deleted: controls/update_def_inputs.cf
deleted: custom-2.cf
deleted: def.json
deleted: inventory/aix.cf
deleted: inventory/any.cf
deleted: inventory/debian.cf
deleted: inventory/freebsd.cf
deleted: inventory/generic.cf
deleted: inventory/linux.cf
deleted: inventory/lsb.cf
deleted: inventory/macos.cf
deleted: inventory/os.cf
deleted: inventory/redhat.cf
deleted: inventory/suse.cf
deleted: inventory/windows.cf
deleted: lib/autorun.cf
deleted: lib/bundles.cf
deleted: lib/cfe_internal.cf
deleted: lib/cfe_internal_hub.cf
deleted: lib/cfengine_enterprise_hub_ha.cf
deleted: lib/commands.cf
deleted: lib/common.cf
deleted: lib/databases.cf
deleted: lib/deprecated-upstream.cf
deleted: lib/edit_xml.cf
deleted: lib/event.cf
deleted: lib/examples.cf
deleted: lib/feature.cf
deleted: lib/files.cf
deleted: lib/guest_environments.cf
deleted: lib/monitor.cf
deleted: lib/packages-ENT-3719.cf
deleted: lib/packages.cf
deleted: lib/paths.cf
deleted: lib/processes.cf
deleted: lib/reports.cf
deleted: lib/services.cf
deleted: lib/stdlib.cf
deleted: lib/storage.cf
deleted: lib/testing.cf
deleted: lib/users.cf
deleted: lib/vcs.cf
deleted: modules/packages/vendored/WiRunSQL.vbs.mustache
deleted: modules/packages/vendored/apk.mustache
deleted: modules/packages/vendored/apt_get.mustache
deleted: modules/packages/vendored/freebsd_ports.mustache
deleted: modules/packages/vendored/msiexec-list.vbs.mustache
deleted: modules/packages/vendored/msiexec.bat.mustache
deleted: modules/packages/vendored/nimclient.mustache
deleted: modules/packages/vendored/pkg.mustache
deleted: modules/packages/vendored/pkgsrc.mustache
deleted: modules/packages/vendored/slackpkg.mustache
deleted: modules/packages/vendored/snap.mustache
deleted: modules/packages/vendored/yum.mustache
deleted: modules/packages/vendored/zypper.mustache
deleted: promises.cf
deleted: services/autorun/custom-1.cf
deleted: services/autorun/hello.cf
deleted: services/custom-3.cf
deleted: services/init.cf
deleted: services/main.cf
deleted: standalone_self_upgrade.cf
deleted: templates/cf-apache.service.mustache
deleted: templates/cf-execd.service.mustache
deleted: templates/cf-hub.service.mustache
deleted: templates/cf-monitord.service.mustache
deleted: templates/cf-postgres.service.mustache
deleted: templates/cf-runalerts.service.mustache
deleted: templates/cf-serverd.service.mustache
deleted: templates/cfengine3.service.mustache
deleted: templates/cfengine_watchdog.mustache
deleted: templates/federated_reporting/10-base_filter.sed
deleted: templates/federated_reporting/50-merge_inserts.awk
deleted: templates/federated_reporting/config.sh.mustache
deleted: templates/federated_reporting/dump.sh
deleted: templates/federated_reporting/import.sh
deleted: templates/federated_reporting/import_file.sh
deleted: templates/federated_reporting/log.sh.mustache
deleted: templates/federated_reporting/parallel.sh
deleted: templates/federated_reporting/psql_wrapper.sh.mustache
deleted: templates/federated_reporting/pull_dumps_from.sh
deleted: templates/federated_reporting/transport.sh
deleted: templates/host_info_report.mustache
deleted: templates/json_multiline.mustache
deleted: templates/json_serial.mustache
deleted: templates/vercmp.ps1
deleted: update.cf
no changes added to commit (use "git add" and/or "git commit -a")
Install the new version of the MPF
Installing from Git
First, clone the desired version of the MPF source.
export MPF_VERSION="3.21.2"
git clone -b $MPF_VERSION https://github.com/cfengine/masterfiles $INTEGRATION_ROOT/masterfiles-source-$MPF_VERSION
Cloning into '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2'...
Note: switching to 'f495603285f9bd90d5d36df4fec4870aeee751e8'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
Then build and install targeting the integration root directory. When installed from source masterfiles installs into the masterfiles
directory.
cd $INTEGRATION_ROOT/masterfiles-source-$MPF_VERSION
export EXPLICIT_VERSION=$MPF_VERSION
./autogen.sh
make
make install prefix=$INTEGRATION_ROOT/
./autogen.sh: Running determine-version.sh ...
./autogen.sh: Running determine-release.sh ...
All tags pointing to current commit:
3.21.2
3.21.2-build4
Latest version: 3.21.2
Could not parse it, using default release number 1
./autogen.sh: Running autoreconf ...
configure.ac:40: installing './config.guess'
configure.ac:40: installing './config.sub'
configure.ac:43: installing './install-sh'
configure.ac:43: installing './missing'
parallel-tests: installing './test-driver'
/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '1000' is supported by ustar format... yes
checking whether GID '1000' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking if GNU tar supports --hard-dereference... yes
checking whether to enable maintainer-specific portions of Makefiles... yes
checking whether make supports nested variables... (cached) yes
checking for pkg_install... no
checking for shunit2... no
Summary:
Version -> 3.21.2
Release -> 1
Core directory -> not set - tests are disabled
Enterprise directory -> not set - some tests are disabled
Install prefix -> /var/cfengine
bindir -> /var/cfengine/bin
configure: generating makefile targets
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating controls/update_def.cf
config.status: creating promises.cf
config.status: creating standalone_self_upgrade.cf
config.status: creating tests/Makefile
config.status: creating tests/acceptance/Makefile
config.status: creating tests/unit/Makefile
DONE: Configuration done. Run "make install" to install CFEngine Masterfiles.
Making all in tests/
make[1]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
Making all in .
make[2]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
make[2]: Nothing to be done for 'all-am'.
make[2]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
Making all in unit
make[2]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests/unit'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests/unit'
make[1]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
make[1]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2'
Making install in tests/
make[1]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
Making install in .
make[2]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
make[3]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
make[3]: Nothing to be done for 'install-exec-am'.
make[3]: Nothing to be done for 'install-data-am'.
make[3]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
make[2]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
Making install in unit
make[2]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests/unit'
make[3]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests/unit'
make[3]: Nothing to be done for 'install-exec-am'.
make[3]: Nothing to be done for 'install-data-am'.
make[3]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests/unit'
make[2]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests/unit'
make[1]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2/tests'
make[1]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2'
make[2]: Entering directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2'
make[2]: Nothing to be done for 'install-exec-am'.
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/core'
/usr/bin/install -c -m 644 ./cfe_internal/core/host_info_report.cf ./cfe_internal/core/log_rotation.cf ./cfe_internal/core/main.cf ./cfe_internal/core/limit_robot_agents.cf '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/core'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/enterprise/templates'
/usr/bin/install -c -m 644 ./cfe_internal/enterprise/templates/runalerts.sh.mustache ./cfe_internal/enterprise/templates/httpd.conf.mustache ./cfe_internal/enterprise/templates/apachectl.mustache ./cfe_internal/enterprise/templates/runalerts.php.mustache '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/enterprise/templates'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/inventory'
/usr/bin/install -c -m 644 ./inventory/windows.cf ./inventory/suse.cf ./inventory/macos.cf ./inventory/lsb.cf ./inventory/any.cf ./inventory/os.cf ./inventory/freebsd.cf ./inventory/generic.cf ./inventory/debian.cf ./inventory/linux.cf ./inventory/redhat.cf ./inventory/aix.cf '/tmp/MPF-upgrade/integration//masterfiles/inventory'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/enterprise/federation'
/usr/bin/install -c -m 644 ./cfe_internal/enterprise/federation/federation.cf '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/enterprise/federation'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/core/deprecated'
/usr/bin/install -c -m 644 ./cfe_internal/core/deprecated/cfengine_processes.cf '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/core/deprecated'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/lib/templates'
/usr/bin/install -c -m 644 ./lib/templates/tap.mustache ./lib/templates/junit.mustache '/tmp/MPF-upgrade/integration//masterfiles/lib/templates'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/services/autorun'
/usr/bin/install -c -m 644 ./services/autorun/hello.cf '/tmp/MPF-upgrade/integration//masterfiles/services/autorun'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/lib'
/usr/bin/install -c -m 644 ./lib/testing.cf ./lib/examples.cf ./lib/packages.cf ./lib/common.cf ./lib/users.cf ./lib/guest_environments.cf ./lib/cfengine_enterprise_hub_ha.cf ./lib/edit_xml.cf ./lib/files.cf ./lib/bundles.cf ./lib/reports.cf ./lib/event.cf ./lib/storage.cf ./lib/paths.cf ./lib/vcs.cf ./lib/stdlib.cf ./lib/autorun.cf ./lib/databases.cf ./lib/feature.cf ./lib/cfe_internal_hub.cf ./lib/monitor.cf ./lib/services.cf ./lib/packages-ENT-3719.cf ./lib/commands.cf ./lib/processes.cf ./lib/cfe_internal.cf '/tmp/MPF-upgrade/integration//masterfiles/lib'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/update'
/usr/bin/install -c -m 644 ./cfe_internal/update/cfe_internal_dc_workflow.cf ./cfe_internal/update/lib.cf ./cfe_internal/update/update_processes.cf ./cfe_internal/update/windows_unattended_upgrade.cf ./cfe_internal/update/systemd_units.cf ./cfe_internal/update/update_policy.cf ./cfe_internal/update/update_bins.cf ./cfe_internal/update/cfe_internal_update_from_repository.cf '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/update'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/controls'
/usr/bin/install -c -m 644 ./controls/cf_agent.cf ./controls/cf_runagent.cf ./controls/cf_execd.cf ./controls/def_inputs.cf ./controls/cf_monitord.cf ./controls/def.cf ./controls/reports.cf ./controls/update_def_inputs.cf ./controls/cf_serverd.cf ./controls/cf_hub.cf ./controls/update_def.cf '/tmp/MPF-upgrade/integration//masterfiles/controls'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/enterprise/ha'
/usr/bin/install -c -m 644 ./cfe_internal/enterprise/ha/ha_def.cf ./cfe_internal/enterprise/ha/ha.cf ./cfe_internal/enterprise/ha/ha_update.cf '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/enterprise/ha'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/modules/packages/vendored'
/usr/bin/install -c -m 644 ./modules/packages/vendored/apk.mustache ./modules/packages/vendored/msiexec.bat.mustache ./modules/packages/vendored/nimclient.mustache ./modules/packages/vendored/snap.mustache ./modules/packages/vendored/yum.mustache ./modules/packages/vendored/msiexec-list.vbs.mustache ./modules/packages/vendored/apt_get.mustache ./modules/packages/vendored/slackpkg.mustache ./modules/packages/vendored/pkgsrc.mustache ./modules/packages/vendored/pkg.mustache ./modules/packages/vendored/freebsd_ports.mustache ./modules/packages/vendored/zypper.mustache ./modules/packages/vendored/WiRunSQL.vbs.mustache '/tmp/MPF-upgrade/integration//masterfiles/modules/packages/vendored'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal'
/usr/bin/install -c -m 644 ./cfe_internal/recommendations.cf ./cfe_internal/CFE_cfengine.cf '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal'
/usr/bin/install -c -m 644 ./update.cf ./promises.cf ./standalone_self_upgrade.cf '/tmp/MPF-upgrade/integration//masterfiles/.'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/core/watchdog'
/usr/bin/install -c -m 644 ./cfe_internal/core/watchdog/watchdog.cf '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/core/watchdog'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/core/watchdog/templates'
/usr/bin/install -c -m 644 ./cfe_internal/core/watchdog/templates/watchdog-windows.ps1.mustache ./cfe_internal/core/watchdog/templates/watchdog.mustache '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/core/watchdog/templates'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/templates'
/usr/bin/install -c -m 644 ./templates/cf-execd.service.mustache ./templates/cf-apache.service.mustache ./templates/host_info_report.mustache ./templates/cf-monitord.service.mustache ./templates/json_serial.mustache ./templates/json_multiline.mustache ./templates/cf-hub.service.mustache ./templates/cfengine3.service.mustache ./templates/cf-postgres.service.mustache ./templates/cfengine_watchdog.mustache ./templates/vercmp.ps1 ./templates/cf-runalerts.service.mustache ./templates/cf-serverd.service.mustache ./templates/cf-reactor.service.mustache '/tmp/MPF-upgrade/integration//masterfiles/templates'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/enterprise'
/usr/bin/install -c -m 644 ./cfe_internal/enterprise/CFE_knowledge.cf ./cfe_internal/enterprise/file_change.cf ./cfe_internal/enterprise/CFE_hub_specific.cf ./cfe_internal/enterprise/mission_portal.cf ./cfe_internal/enterprise/main.cf '/tmp/MPF-upgrade/integration//masterfiles/cfe_internal/enterprise'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/templates/federated_reporting'
/usr/bin/install -c -m 644 ./templates/federated_reporting/cfsecret.py ./templates/federated_reporting/import_file.sh ./templates/federated_reporting/psql_wrapper.sh.mustache ./templates/federated_reporting/import.sh ./templates/federated_reporting/transfer_distributed_cleanup_items.sh ./templates/federated_reporting/config.sh.mustache ./templates/federated_reporting/distributed_cleanup.py ./templates/federated_reporting/transport.sh ./templates/federated_reporting/log.sh.mustache ./templates/federated_reporting/dump.sh ./templates/federated_reporting/10-base_filter.sed ./templates/federated_reporting/nova_api.py ./templates/federated_reporting/pull_dumps_from.sh ./templates/federated_reporting/50-merge_inserts.awk ./templates/federated_reporting/parallel.sh '/tmp/MPF-upgrade/integration//masterfiles/templates/federated_reporting'
/usr/bin/mkdir -p '/tmp/MPF-upgrade/integration//masterfiles/services'
/usr/bin/install -c -m 644 ./services/init.cf ./services/main.cf '/tmp/MPF-upgrade/integration//masterfiles/services'
make[2]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2'
make[1]: Leaving directory '/tmp/MPF-upgrade/integration/masterfiles-source-3.21.2'
We no longer need the source, we can clean it up.
cd $INTEGRATION_ROOT/
rm -rf $INTEGRATION_ROOT/masterfiles-source-$MPF_VERSION
Merge differences
Now we can use git status
to see an overview of the changes to the repository between our starting point and the new MPF.
cd $INTEGRATION_ROOT/masterfiles
git status
On branch master
Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: cfe_internal/core/watchdog/templates/watchdog.mustache
modified: cfe_internal/enterprise/CFE_hub_specific.cf
modified: cfe_internal/enterprise/CFE_knowledge.cf
modified: cfe_internal/enterprise/federation/federation.cf
modified: cfe_internal/enterprise/file_change.cf
modified: cfe_internal/enterprise/main.cf
modified: cfe_internal/enterprise/mission_portal.cf
modified: cfe_internal/enterprise/templates/httpd.conf.mustache
modified: cfe_internal/update/cfe_internal_dc_workflow.cf
modified: cfe_internal/update/cfe_internal_update_from_repository.cf
modified: cfe_internal/update/lib.cf
modified: cfe_internal/update/update_bins.cf
modified: cfe_internal/update/update_policy.cf
modified: cfe_internal/update/update_processes.cf
modified: cfe_internal/update/windows_unattended_upgrade.cf
modified: controls/cf_agent.cf
modified: controls/cf_execd.cf
modified: controls/cf_serverd.cf
modified: controls/def.cf
modified: controls/reports.cf
modified: controls/update_def.cf
deleted: custom-2.cf
deleted: def.json
modified: inventory/any.cf
modified: inventory/debian.cf
modified: inventory/linux.cf
modified: inventory/os.cf
modified: inventory/redhat.cf
modified: lib/autorun.cf
modified: lib/bundles.cf
modified: lib/cfe_internal_hub.cf
deleted: lib/deprecated-upstream.cf
modified: lib/files.cf
modified: lib/packages.cf
modified: lib/paths.cf
modified: lib/services.cf
modified: modules/packages/vendored/apt_get.mustache
modified: modules/packages/vendored/msiexec-list.vbs.mustache
modified: modules/packages/vendored/nimclient.mustache
modified: modules/packages/vendored/pkg.mustache
modified: modules/packages/vendored/zypper.mustache
modified: promises.cf
deleted: services/autorun/custom-1.cf
deleted: services/custom-3.cf
modified: services/main.cf
modified: standalone_self_upgrade.cf
modified: templates/cf-apache.service.mustache
modified: templates/cf-execd.service.mustache
modified: templates/cf-hub.service.mustache
modified: templates/cf-monitord.service.mustache
modified: templates/cf-postgres.service.mustache
modified: templates/cf-runalerts.service.mustache
modified: templates/cf-serverd.service.mustache
modified: templates/federated_reporting/config.sh.mustache
modified: templates/federated_reporting/dump.sh
modified: templates/federated_reporting/import.sh
modified: templates/federated_reporting/psql_wrapper.sh.mustache
modified: templates/federated_reporting/pull_dumps_from.sh
modified: update.cf
Untracked files:
(use "git add <file>..." to include in what will be committed)
cfe_internal/enterprise/templates/apachectl.mustache
lib/templates/
templates/cf-reactor.service.mustache
templates/federated_reporting/cfsecret.py
templates/federated_reporting/distributed_cleanup.py
templates/federated_reporting/nova_api.py
templates/federated_reporting/transfer_distributed_cleanup_items.sh
no changes added to commit (use "git add" and/or "git commit -a")
All of the Untracked files are new additions from upstream so they should be safe to take.
git add cfe_internal/enterprise/templates/apachectl.mustache
git add lib/templates/junit.mustache
git add lib/templates/tap.mustache
git add templates/cf-reactor.service.mustache
git add templates/federated_reporting/cfsecret.py
git add templates/federated_reporting/distributed_cleanup.py
git add templates/federated_reporting/nova_api.py
git add templates/federated_reporting/transfer_distributed_cleanup_items.sh
We can run git status again to see the current overview:
git status
On branch master
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: cfe_internal/enterprise/templates/apachectl.mustache
new file: lib/templates/junit.mustache
new file: lib/templates/tap.mustache
new file: templates/cf-reactor.service.mustache
new file: templates/federated_reporting/cfsecret.py
new file: templates/federated_reporting/distributed_cleanup.py
new file: templates/federated_reporting/nova_api.py
new file: templates/federated_reporting/transfer_distributed_cleanup_items.sh
Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: cfe_internal/core/watchdog/templates/watchdog.mustache
modified: cfe_internal/enterprise/CFE_hub_specific.cf
modified: cfe_internal/enterprise/CFE_knowledge.cf
modified: cfe_internal/enterprise/federation/federation.cf
modified: cfe_internal/enterprise/file_change.cf
modified: cfe_internal/enterprise/main.cf
modified: cfe_internal/enterprise/mission_portal.cf
modified: cfe_internal/enterprise/templates/httpd.conf.mustache
modified: cfe_internal/update/cfe_internal_dc_workflow.cf
modified: cfe_internal/update/cfe_internal_update_from_repository.cf
modified: cfe_internal/update/lib.cf
modified: cfe_internal/update/update_bins.cf
modified: cfe_internal/update/update_policy.cf
modified: cfe_internal/update/update_processes.cf
modified: cfe_internal/update/windows_unattended_upgrade.cf
modified: controls/cf_agent.cf
modified: controls/cf_execd.cf
modified: controls/cf_serverd.cf
modified: controls/def.cf
modified: controls/reports.cf
modified: controls/update_def.cf
deleted: custom-2.cf
deleted: def.json
modified: inventory/any.cf
modified: inventory/debian.cf
modified: inventory/linux.cf
modified: inventory/os.cf
modified: inventory/redhat.cf
modified: lib/autorun.cf
modified: lib/bundles.cf
modified: lib/cfe_internal_hub.cf
deleted: lib/deprecated-upstream.cf
modified: lib/files.cf
modified: lib/packages.cf
modified: lib/paths.cf
modified: lib/services.cf
modified: modules/packages/vendored/apt_get.mustache
modified: modules/packages/vendored/msiexec-list.vbs.mustache
modified: modules/packages/vendored/nimclient.mustache
modified: modules/packages/vendored/pkg.mustache
modified: modules/packages/vendored/zypper.mustache
modified: promises.cf
deleted: services/autorun/custom-1.cf
deleted: services/custom-3.cf
modified: services/main.cf
modified: standalone_self_upgrade.cf
modified: templates/cf-apache.service.mustache
modified: templates/cf-execd.service.mustache
modified: templates/cf-hub.service.mustache
modified: templates/cf-monitord.service.mustache
modified: templates/cf-postgres.service.mustache
modified: templates/cf-runalerts.service.mustache
modified: templates/cf-serverd.service.mustache
modified: templates/federated_reporting/config.sh.mustache
modified: templates/federated_reporting/dump.sh
modified: templates/federated_reporting/import.sh
modified: templates/federated_reporting/psql_wrapper.sh.mustache
modified: templates/federated_reporting/pull_dumps_from.sh
modified: update.cf
Next we want to bring back any of our custom files. Look through the deleted files, identify your custom files and restore them with git checkout
.
git ls-files --deleted
custom-2.cf
def.json
lib/deprecated-upstream.cf
services/autorun/custom-1.cf
services/custom-3.cf
Keeping your polices organized together helps to make this process easy. The custom policy files in the example policy set are def.json
, services/autorun/custom-1.cf
, custom-2.cf
, and services/custom-3.cf
.
git checkout custom-2.cf
git checkout def.json
git checkout services/autorun/custom-1.cf
git checkout services/custom-3.cf
Updated 1 path from the index
Updated 1 path from the index
Updated 1 path from the index
Updated 1 path from the index
Other deleted files from the upstream framework like lib/deprecated-upstream.cf
should be deleted with git rm
.
Note: It is uncommon for any files to be moved or deleted between patch releases (e.g. 3.18.0
-> 3.18.5
) like lib/deprecated-upstream.cf
in this example.
git rm lib/deprecated-upstream.cf
rm 'lib/deprecated-upstream.cf'
The files marked as modified in the git status
output are files that have changed upstream.
git status
On branch master
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: cfe_internal/enterprise/templates/apachectl.mustache
deleted: lib/deprecated-upstream.cf
new file: lib/templates/junit.mustache
new file: lib/templates/tap.mustache
new file: templates/cf-reactor.service.mustache
new file: templates/federated_reporting/cfsecret.py
new file: templates/federated_reporting/distributed_cleanup.py
new file: templates/federated_reporting/nova_api.py
new file: templates/federated_reporting/transfer_distributed_cleanup_items.sh
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: cfe_internal/core/watchdog/templates/watchdog.mustache
modified: cfe_internal/enterprise/CFE_hub_specific.cf
modified: cfe_internal/enterprise/CFE_knowledge.cf
modified: cfe_internal/enterprise/federation/federation.cf
modified: cfe_internal/enterprise/file_change.cf
modified: cfe_internal/enterprise/main.cf
modified: cfe_internal/enterprise/mission_portal.cf
modified: cfe_internal/enterprise/templates/httpd.conf.mustache
modified: cfe_internal/update/cfe_internal_dc_workflow.cf
modified: cfe_internal/update/cfe_internal_update_from_repository.cf
modified: cfe_internal/update/lib.cf
modified: cfe_internal/update/update_bins.cf
modified: cfe_internal/update/update_policy.cf
modified: cfe_internal/update/update_processes.cf
modified: cfe_internal/update/windows_unattended_upgrade.cf
modified: controls/cf_agent.cf
modified: controls/cf_execd.cf
modified: controls/cf_serverd.cf
modified: controls/def.cf
modified: controls/reports.cf
modified: controls/update_def.cf
modified: inventory/any.cf
modified: inventory/debian.cf
modified: inventory/linux.cf
modified: inventory/os.cf
modified: inventory/redhat.cf
modified: lib/autorun.cf
modified: lib/bundles.cf
modified: lib/cfe_internal_hub.cf
modified: lib/files.cf
modified: lib/packages.cf
modified: lib/paths.cf
modified: lib/services.cf
modified: modules/packages/vendored/apt_get.mustache
modified: modules/packages/vendored/msiexec-list.vbs.mustache
modified: modules/packages/vendored/nimclient.mustache
modified: modules/packages/vendored/pkg.mustache
modified: modules/packages/vendored/zypper.mustache
modified: promises.cf
modified: services/main.cf
modified: standalone_self_upgrade.cf
modified: templates/cf-apache.service.mustache
modified: templates/cf-execd.service.mustache
modified: templates/cf-hub.service.mustache
modified: templates/cf-monitord.service.mustache
modified: templates/cf-postgres.service.mustache
modified: templates/cf-runalerts.service.mustache
modified: templates/cf-serverd.service.mustache
modified: templates/federated_reporting/config.sh.mustache
modified: templates/federated_reporting/dump.sh
modified: templates/federated_reporting/import.sh
modified: templates/federated_reporting/psql_wrapper.sh.mustache
modified: templates/federated_reporting/pull_dumps_from.sh
modified: update.cf
It's best to review the diff of each modified file to understand the upstream changes as well as identify any local modifications that need to be retained. You should always keep a good record of any modifications made to vendored files to ensure that nothing is lost during future framework upgrades.
For example, here the diff for promises.cf
shows upstream changes but also highlights where the vendored policy had been customized to integrate a custom policy.
git diff promises.cf
Output:
diff --git a/promises.cf b/promises.cf
index 15c0c40..4611098 100644
+++ b/def.json
@@ -1,8 +1,11 @@
{
- "inputs": [ "services/custom-3.cf" ],
+ "inputs": [ "custom-2.cf", "services/custom-3.cf" ],
"classes": {
"default:services_autorun": {
"class_expressions": [ "any::" ],
"comment": "We want to use the autorun functionality because it is convenient."
- }
+ },
+ "vars":{
+ "control_common_bundlesequence_end": [ "custom_2" ]
+ }
}
\ No newline at end of file
So, we now want to accept all the changes to promises.cf
and def.json
.
git add promises.cf def.json
If you are unsure if or how to integrate customizations without modifying vendored policy reach out to support for help. For any modified files that you have not customized simply stage them for commit with git add
.
git add cfe_internal/core/watchdog/templates/watchdog.mustache
git add cfe_internal/enterprise/CFE_hub_specific.cf
git add cfe_internal/enterprise/CFE_knowledge.cf
git add cfe_internal/enterprise/federation/federation.cf
git add cfe_internal/enterprise/file_change.cf
git add cfe_internal/enterprise/main.cf
git add cfe_internal/enterprise/mission_portal.cf
git add cfe_internal/enterprise/templates/httpd.conf.mustache
git add cfe_internal/update/cfe_internal_dc_workflow.cf
git add cfe_internal/update/cfe_internal_update_from_repository.cf
git add cfe_internal/update/lib.cf
git add cfe_internal/update/update_bins.cf
git add cfe_internal/update/update_policy.cf
git add cfe_internal/update/update_processes.cf
git add cfe_internal/update/windows_unattended_upgrade.cf
git add controls/cf_agent.cf
git add controls/cf_execd.cf
git add controls/cf_serverd.cf
git add controls/def.cf
git add controls/reports.cf
git add controls/update_def.cf
git add def.json
git add inventory/any.cf
git add inventory/debian.cf
git add inventory/linux.cf
git add inventory/os.cf
git add inventory/redhat.cf
git add lib/autorun.cf
git add lib/bundles.cf
git add lib/cfe_internal_hub.cf
git add lib/files.cf
git add lib/packages.cf
git add lib/paths.cf
git add lib/services.cf
git add modules/packages/vendored/apt_get.mustache
git add modules/packages/vendored/msiexec-list.vbs.mustache
git add modules/packages/vendored/nimclient.mustache
git add modules/packages/vendored/pkg.mustache
git add modules/packages/vendored/zypper.mustache
git add promises.cf
git add services/main.cf
git add standalone_self_upgrade.cf
git add templates/cf-apache.service.mustache
git add templates/cf-execd.service.mustache
git add templates/cf-hub.service.mustache
git add templates/cf-monitord.service.mustache
git add templates/cf-postgres.service.mustache
git add templates/cf-runalerts.service.mustache
git add templates/cf-serverd.service.mustache
git add templates/federated_reporting/config.sh.mustache
git add templates/federated_reporting/dump.sh
git add templates/federated_reporting/import.sh
git add templates/federated_reporting/psql_wrapper.sh.mustache
git add templates/federated_reporting/pull_dumps_from.sh
git add update.cf
Review git status
one more time to make sure the changes are as expected.
git status
On branch master
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: cfe_internal/core/watchdog/templates/watchdog.mustache
modified: cfe_internal/enterprise/CFE_hub_specific.cf
modified: cfe_internal/enterprise/CFE_knowledge.cf
modified: cfe_internal/enterprise/federation/federation.cf
modified: cfe_internal/enterprise/file_change.cf
modified: cfe_internal/enterprise/main.cf
modified: cfe_internal/enterprise/mission_portal.cf
new file: cfe_internal/enterprise/templates/apachectl.mustache
modified: cfe_internal/enterprise/templates/httpd.conf.mustache
modified: cfe_internal/update/cfe_internal_dc_workflow.cf
modified: cfe_internal/update/cfe_internal_update_from_repository.cf
modified: cfe_internal/update/lib.cf
modified: cfe_internal/update/update_bins.cf
modified: cfe_internal/update/update_policy.cf
modified: cfe_internal/update/update_processes.cf
modified: cfe_internal/update/windows_unattended_upgrade.cf
modified: controls/cf_agent.cf
modified: controls/cf_execd.cf
modified: controls/cf_serverd.cf
modified: controls/def.cf
modified: controls/reports.cf
modified: controls/update_def.cf
modified: def.json
modified: inventory/any.cf
modified: inventory/debian.cf
modified: inventory/linux.cf
modified: inventory/os.cf
modified: inventory/redhat.cf
modified: lib/autorun.cf
modified: lib/bundles.cf
modified: lib/cfe_internal_hub.cf
deleted: lib/deprecated-upstream.cf
modified: lib/files.cf
modified: lib/packages.cf
modified: lib/paths.cf
modified: lib/services.cf
new file: lib/templates/junit.mustache
new file: lib/templates/tap.mustache
modified: modules/packages/vendored/apt_get.mustache
modified: modules/packages/vendored/msiexec-list.vbs.mustache
modified: modules/packages/vendored/nimclient.mustache
modified: modules/packages/vendored/pkg.mustache
modified: modules/packages/vendored/zypper.mustache
modified: promises.cf
modified: services/main.cf
modified: standalone_self_upgrade.cf
modified: templates/cf-apache.service.mustache
modified: templates/cf-execd.service.mustache
modified: templates/cf-hub.service.mustache
modified: templates/cf-monitord.service.mustache
modified: templates/cf-postgres.service.mustache
new file: templates/cf-reactor.service.mustache
modified: templates/cf-runalerts.service.mustache
modified: templates/cf-serverd.service.mustache
new file: templates/federated_reporting/cfsecret.py
modified: templates/federated_reporting/config.sh.mustache
new file: templates/federated_reporting/distributed_cleanup.py
modified: templates/federated_reporting/dump.sh
modified: templates/federated_reporting/import.sh
new file: templates/federated_reporting/nova_api.py
modified: templates/federated_reporting/psql_wrapper.sh.mustache
modified: templates/federated_reporting/pull_dumps_from.sh
new file: templates/federated_reporting/transfer_distributed_cleanup_items.sh
modified: update.cf
Make sure the policy validates and commit your changes.
git commit -m "Upgraded MPF from 3.18.0 to 3.21.2"
[master a5d512c] Upgraded MPF from 3.18.0 to 3.21.2
64 files changed, 2599 insertions(+), 728 deletions(-)
create mode 100644 cfe_internal/enterprise/templates/apachectl.mustache
rewrite inventory/redhat.cf (63%)
delete mode 100644 lib/deprecated-upstream.cf
create mode 100644 lib/templates/junit.mustache
create mode 100644 lib/templates/tap.mustache
create mode 100644 templates/cf-reactor.service.mustache
create mode 100644 templates/federated_reporting/cfsecret.py
create mode 100644 templates/federated_reporting/distributed_cleanup.py
create mode 100644 templates/federated_reporting/nova_api.py
create mode 100644 templates/federated_reporting/transfer_distributed_cleanup_items.sh
Now your Masterfiles Policy Framework is upgraded and ready to be tested.
Tags for variables, classes, and bundles
Introduction
meta tags can be attached to any promise type using the meta
attribute.
These tags are useful for cross-referencing related promises. bundles
, vars
and classes
can be identified and leveraged in different ways within policy
using these tags.
Problem statement
We'd like to apply tags to variables and classes for many purposes, from stating their provenance (whence they came, why they exist, and how they can be used) to filtering them based on tags.
We'd also like to be able to include all the files in a directory and then run all the discovered bundles if they are tagged appropriately.
Syntax
Tagging variables and classes is easy with the meta
attribute. Here's an
example that sets the inventory
tag on a variable and names the attribute that
it represents. This one is actually built into the standard
MPF inventory policy,
so it's available out of the box in either Community or Enterprise.
bundle agent cfe_autorun_inventory_listening_ports
{
vars:
"ports" -> { "ENT-150" }
slist => sort( "mon.listening_ports", "int"),
meta => { "inventory", "attribute_name=Ports listening" },
if => some("[0-9]+", "mon.listening_ports"),
comment => "We only want to inventory the listening ports if we have
values that make sense.";
}
In the Enterprise Mission Portal, you can then make a report for "Ports listening" across all your machines. For more details, see Enterprise reporting
Class tags work exactly the same way, you just apply them to a
classes
promise with the meta
attribute.
Tagging bundles is different because you have to use the meta
promise type (different from the meta
attribute).
An example is easiest:
bundle agent run_deprecated
{
meta:
"tags" slist => { "deprecated" };
}
This declares an agent bundle with a single tag.
Functions
Several new functions exist to give you access to variable and class tags, and to find classes and variables with tags.
classesmatching
: this used to be somewhat available with theallclasses.txt
file. You can now call a function to get all the defined classes, optionally filtering by name and tags. See classesmatchinggetvariablemetatags
: get the tags of a variable as an slist. See getvariablemetatagsvariablesmatching
: just likeclassesmatching
but for variables. See variablesmatchingvariablesmatching_as_data
: likevariablesmatching
but the matching variables and values are returned as a merged data container. See variablesmatching_as_datagetclassmetatags
: get the tags of a class as an slist. See getclassmetatagsbundlesmatching
: find the bundles matching some tags. See bundlesmatching (the example shows how you'd find adeprecated
bundle likerun_deprecated
earlier).
Module protocol
The module protocol has been extended to support tags. You set the tags on a line and they persist for every subsequent variable or class.
^meta=inventory
+x
=a=100
^meta=report,attribute_name=My vars
+y
=n=100
This will create class x
and variable a
with tag inventory
.
Then it will create class y
and variable b
with tags report
and
attribute_name=My vars
.
Enterprise reporting with tags
In CFEngine Enterprise, you can build reports based on tagged variables and classes.
Please see Enterprise reporting for a full tutorial, including troubleshooting possible errors. In short, this is an extremely easy way to categorize various data accessible to the agent.
Dynamic bundlesequence
Dynamic bundlesequences are extremely easy. First you find all the bundles whos name matches a regular expression and N tags.
vars:
"bundles" slist => bundlesmatching("regex", "tag1", "tag2", ...);
Then every bundle matching the regular expression regex
and all
the tags will be found and run.
methods:
"run $(bundles)" usebundle => $(bundles);
Note that the discovered bundle names will have the namespace prefix,
e.g. default:mybundle
. The regular expression has to match that. So
mybundle
as the regular expression would not work. See
bundlesmatching
for another detailed example.
In fact we found this so useful we implemented services autorun in the masterfiles policy framework.
There is only one thing to beware. All the bundles have to have the same number of arguments (0 in the case shown). Otherwise you will get a runtime error and CFEngine will abort. We recommend only using 0-argument bundles in a dynamic sequence to reduce this risk.
Summary
Tagging variables and classes and bundles in CFEngine is easy and allows more dynamic behavior than ever before. Try it out and see for yourself how it will change the way you use and think about system configuration policy and CFEngine.
Custom inventory
This tutorial will show you how to add custom inventory attributes that can be leveraged in policy and reported on in the CFEngine Enterprise Mission Portal. For a more detailed overview on how the inventory system works please reference CFEngine 3 inventory modules.
Overview
This tutorial provides instructions for the following:
Note: This tutorial uses the CFEngine Enterprise Vagrant Environment and files located in the vagrant project directory are automatically available to all hosts.
Choose an attribute to inventory
Writing inventory policy is incredibly easy. Simply add the inventory
and
attribute_name=
tags to any variable or namespace scoped classes.
In this tutorial we will add Owner
information into the inventory. In this
example we will use a simple shared flat file data source
/vagrant/inventory_owner.csv
.
On your hosts create /vagrant/inventory_owner.csv
with the following content:
hub, Operations Team <ops@cfengine.com>
host001, Development <dev@cfengine.com>
Create and deploy inventory policy
Now that each of your hosts has access to a data source that provides the Owner information we will write an inventory policy to report that information.
Create /var/cfengine/masterfiles/services/tutorials/inventory/owner.cf
with the
following content:
bundle agent tutorials_inventory_owner
# @brief Inventory Owner information
# @description Inventory owner information from `/vagrant/inventory_owner.csv`.
{
vars:
"data_source" string => "/vagrant/inventory_owner.csv";
"owners"
data => data_readstringarray( $(data_source), "", ", ", 100, 512 ),
if => fileexists( $(data_source) );
"my_owner"
string => "$(owners[$(sys.uqhost)][0])",
meta => { "inventory", "attribute_name=Owner" },
comment => "We need to tag the owner information so that it is correctly
reported via the UI.";
reports:
inform_mode::
"$(this.bundle): Discovered Owner='$(my_owner)'"
if => isvaribale( "my_owner" );
}
bundle agent __main__
# @brief Run tutorials_inventory_owner if this policy file is the entry
{
methods: "tutorials_inventory_owner";
}
Note: For the simplicity of this tutorial we assume that masterfiles is not configured for policy updates from a Git repository. If it is, please add the policy to your repository and ensure it gets to its final destination as needed.
This policy will not be activated until it has been included in
inputs. For simplicity we will be
adding it via Augments (def.json
).
Create /var/cfengine/masterfiles/def.json
and populate it with the following content:
{
"inputs": [ "services/tutorials/inventory/owner.cf" ],
"vars": {
"control_common_bundlesequence_end": [ "tutorials_inventory_owner" ]
}
}
Any time you modify something, it is always a good idea to validate the syntax. You can run cf-promises
to check policy syntax.
Policy Validation:
[root@hub ~]# cf-promises -cf /var/cfengine/masterfiles/promises.cf
[root@hub ~]# echo $?
0
No output and return code 0
indicate the policy was successfully validated.
JSON Validation:
You can use your favorite JSON validate. I like jq
, plus it's handy for picking apart API responses so let's install that and use it.
[root@hub ~]# wget -q -O /var/cfengine/bin/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
[root@hub ~]# chmod +x /var/cfengine/bin/jq
Once it's installed, we can use it to validate our JSON.
[root@hub ~]# jq '.' < /var/cfengine/masterfiles/def.json
{
"inputs": [
"services/tutorials/inventory/owner.cf"
],
"vars": {
"control_common_bundlesequence_end": [
"tutorials_inventory_owner"
]
}
}
[root@hub ~]# echo $?
0
Pretty printed JSON and a return code of 0
indicate the JSON was successfully validated.
You can also perform a manual policy run and check that the correct owner is discovered.
Manual Policy Run:
cf-agent -KIf /var/cfengine/masterfiles/promises.cf -b tutorials_inventory_owner
info: Using command line specified bundlesequence
R: tutorials_inventory_owner: Discovered Owner='Operations Team <ops@cfengine.com>'
Here we ran the policy without locks (-K
) in inform mode (-I
), using a
specific policy entry (-f
) and activating only a specific bundle (-b
). The
inform output helps us confirm that the owner is discovered from our CSV
properly.
Reporting
Once you have integrated the policy into def.json
it will run by all agents
after they have updated their policy. Once the hub has had a chance to collect
reports the Owner
attribute will be available to select as a Table column for
Inventory reports. Custom attributes appear under the User defined
section.
Note: It may take up to 15 minutes for your custom inventory attributes to be collected and made available for reporting.
Mission Portal
You will see the host owner as shown in the following report.
Inventory API
Of course, you can also get this information from the Inventory API.
Let's query the API from the hub itself, and use jq
to make it easier to handle the output.
Now that we have jq in place, let's query the Inventory API to see what inventory attributes are available.
curl -s -k --user admin:admin -X GET https://localhost/api/inventory/attributes-dictionary | jq '.[].attribute_name'
"Architecture"
"BIOS vendor"
"BIOS version"
"CFEngine Enterprise license file"
"CFEngine Enterprise license status"
"CFEngine Enterprise license utilization"
"CFEngine Enterprise licenses allocated"
"CFEngine ID"
"CFEngine roles"
"CFEngine version"
"CPU logical cores"
"CPU model"
"CPU physical cores"
"CPU sockets"
"Disk free (%)"
"Host name"
"IPv4 addresses"
"Interfaces"
"MAC addresses"
"Memory size (MB)"
"OS"
"OS kernel"
"OS type"
"Owner"
"Physical memory (MB)"
"Policy Release Id"
"Policy Servers"
"Ports listening"
"Primary Policy Server"
"System UUID"
"System manufacturer"
"System product name"
"System serial number"
"System version"
"Timezone"
"Timezone GMT Offset"
"Uptime minutes"
Yes, we can see our attribute Owner
is reported.
Now, let's query the Inventory API to see what Owners are reported.
curl -s -k --user admin:admin -X POST -H 'content-type: application/json' -d '{ "select": [ "Host name", "Owner" ]}' https://localhost/api/inventory | jq '.data[].rows[]'
[
"host001.example.com",
"Development <dev@cfengine.com>"
]
[
"hub.example.com",
"Operations Team <ops@cfengine.com>"
]
Indeed, we can see each host reporting the values as expected from our CSV file.
Dashboard alerts
At 5 minutes intervals, the CFEngine hub gathers information from all of its connected agents about the current state of the system, including the outcome of its runs. All of this information is available to you. In this tutorial we will show how to use the Dashboard to create compliance overview at a glance
Note: This tutorial builds upon another tutorial that manages local users.
We will create 3 alerts, one that shows when CFEngine repairs the system (promise repaired), one that shows when CFEngine does not need to make a change (promise kept), and one that shows CFEngine failing to repair the system (promise not kept).
Create a new alert
Log into Mission Portal, the web interface for CFEngine Enterprise.
In an empty space, Click Add.
Name the alert 'Users compliance' and leave Severity-level at 'medium'.
Click create new condition and leave Name 'Users compliance'.
Select Type to 'Policy'.
Select Filter to 'Bundle', and type 'ensure_users'.
Type Promise handle to 'ensure_user_setup'.
Type Promise Status to 'Not kept'.
- Press 'Save button' and give the Widget an descriptive name.
Done!
You have created a Dashboard Alert for our user management policy. Whenever the policy is out of compliance (not kept), we will be notified.
On the first screen below, we see that the alert has been triggered on zero of our three hosts (0 / 3). This is exactly what we want. There is no promise not kept anywhere, which means we are in compliance.
If you click on the Dashboard tab and go to the front page, you will see that our User Policy has a green check-mark. This means that the 'not kept' condition have not occured on any host.
- Conclusions
In this tutorial, we have shown how easy it is to prove compliance of any of your policies by using the Dashboard alert functionality.
If you would like to get an overview of whenever CFEngine is making a change to your system, simply create another alert, but this time set the Promise Status to 'Repaired'. This time you will see an alert whenever CFEngine is repairing a drift, for instance if a user is accidentially deleted.
Integrating alerts with PagerDuty
In this How To tutorial we will show you can integrate with PagerDuty using the CFEngine notification dashboard.
We will create a policy that ensures file integrity, and have CFEngine notify PagerDuty whenever there is a change in the file we manage.
System requirements:
- CFEngine Mission Portal
- Active PagerDuty Account
Create the file we want to manage
Run the following command on your policy server to create the file we want to manage.
touch /tmp/file-integrity
Create a new policy to manage the file
Insert the following policy into /tmp/file_example.cf
bundle agent file_integrity
{
files:
any::
"/tmp/test-integrity" -> {"PCI-DSS-2", "SOX-nightmare"}
handle => "ensure-test-file-integrity",
changes => change_detection;
}
body changes change_detection
{
hash => "md5";
update_hashes => "true";
report_changes => "all";
report_diffs => "true";
}
Ensure the policy always runs
Normally, to ensure your policy file is put into action, you would need to follow these three steps:
Move the policy file to your masterfiles directory (
/var/cfengine/masterfiles
):Normally, to ensure your policy file is put into action, you would need to follow these three steps:
commandmv /tmp/file_example.cf /var/cfengine/masterfiles/
Modify
promises.cf
to include your policyUnless you use version control system, or has a non-standard CFEngine setup, modify your
promises.cf
file by adding the new bundlename and policy-file so it will be picked up by CFEngine to be included in all future runs.commandvi /var/cfengine/masterfiles/promises.cf
a) Under the body common control, add
file_integrity
to your bundlesequenceb) Under
body common control
, addfile_example.cf
to your inputs section.Now, any change you manually make to the
/tmp/file_integrity
file will be picked up by CFEngine!Next we need to a new service in PagerDuty which we will notify whenever a change is detected by CFEngine.
Create a new service in PagerDuty
Go to PagerDuty.com. In your account, under Services tab, click
Add New Service
Enter a name for the service and select an escalation policy. Select
Integrate via email.
Copy the integration email provided for use in CFEngine.Click
Add Service
button. Copy the integration email which we will use in CFEngine.
Create a new alert in CFEngine Mission Portal
Go to the the CFEngine Dashboard and click
Add
button to create a new alert.Fill out a new alert name
File integrity demo
, severity levelHigh
and name for the conditionFile integrity demo
.Select
Policy
under typeSelect
Bundle
, type in the bundle name which is file_integrity, and finally selectRepaired
as the promise status. This means that whenever CFEngine needs to repair the bundle, it will create an alert notification.Type in the integration email defined above in the Notifications section. Press
Save
to active the alert. Choose any name you like for the New widget. In our demo we name the widgetPagerDuty
.Integration complete!
Test it!
Now we have a made a policy to monitor the /tmp/file-integrity
file. Whenever there is a change to this file, whether it be permissions or content, this will be detected by CFEngine which will send a notification to PagerDuty.
Make a change to the
/tmp/file_integrity
file on your policy server:commandecho "Hello World!!" > /tmp/file_integrity
The next time CFEngine runs, it will detect the change and send an notification to PagerDuty. Go to PagerDuty and wait for an alert to be triggered.
Integrating alerts with ticketing systems
Custom actions can be used to integrate with external 3rd party systems. This tutorial shows how to use a custom action script to open a ticket in Jira when a condition is observed.
How it works
We assume that there is already a CFEngine policy bundle in place called web_service
that ensures an important service is working.
As we are already using the JIRA ticketing system to get notified about issues with the infrastructure, we want CFEngine to open a ticket if our web_service
bundle fails (is not kept). This is done centrally on our hub, because it knows the outcome of the policy on all the nodes. We will only open a ticket once the alert changes state to triggered, not while it remains in triggered, to avoid an unnecessary amount of tickets being automatically created.
Note however that it is possible to expand on this by adjusting the Custom action script. For example, we could create reminder tickets, or even automatically close tickets when the alert clears.
Create a custom action script that creates a new ticket
Log in to the console of your CFEngine hub, and make sure you have python and the jira python package installed (normally by running
pip install jira
).On your workstation, unpack cfengine_custom_action_jira.py to a working directory.
Inside the script, fill in
MYJIRASERVER
,MYUSERNAME
andMYPASSWORD
with your information.Test the script by unpacking alert_parameters_test into the same directory and running
./cfengine_custom_action_jira.py alert_parameters
.Verify the previous step created a ticket in JIRA. If not, recheck the information to typed in, connectivity and any output generated when running the script.
Upload the custom action script to the Mission Portal
Log in to the Mission Portal of CFEngine, go to Settings (top right) followed by Custom notification scripts.
Click on the button to Add a script, upload the script and fill in the information as shown in the screenshot.
Click save to allow the script to be used when creating alerts.
Create a new alert and associate the custom action script
Log into the Mission Portal of CFEngine, click the Dashboard tab.
Click on the existing Policy compliance widget, followed by Add alert.
Name the alert "
Web service
" and setSeverity-level
at "high
".Click create new condition and leave Name "
Web service
".Select Type to "
Policy
".Select Filter to "
Bundle
", and type "web_service
".Type Promise Status to "
Not kept
".Associate the Custom action script we uploaded with the alert.
Conclusions
In this tutorial, we have shown how easy it is to integrate with a ticketing system, with JIRA as an example, using CFEngine Custom actions scripts.
Using this Custom action, you can choose to open JIRA tickets when some or all of your alerts are triggered. But this is just the beginning; using Custom actions, you can integrate with virtually any external system for notifying about- or handling triggered alerts.
Read more in the Custom action documentation.
Integrating with Sumo Logic
In this How To we will show a simple integrate with Sumo Logic. Whenever there is a CFEngine policy update, that event will be exported to Sumo Logic. These events can become valuable traces when using Sumo Logic to analyze and detect unintendent system behavior.
Requirements:
- CFEngine Community/Enterprise
- Sumo Logic account (secret URL)
How it works
Whenever there is a policy update or a new policy is detected by CFEngine, a special variable called "sys.last_policy_update
" will be updated with current timestamp.
We will store this timestamp in a file, and then via api upload the file to Sumo Logic.
Create the CFEngine policy file
In this section we will explain the most important parts of our policy file.
First, we define a couple of variables.
policy_udpate_file
is the variable that contains the name of the file where we will store the timestamp and eventually upload to Sumo Logic.
The two Sumo variables are used to access the service, while the curl_args
is the actual curl command that will upload our timestamp file to Sumo Logic.
vars:
"policy_update_file"
string => "/tmp/CFEngine_policy_updated";
"sumo_url"
string => "https://collectors.sumologic.com/receiver/v1/http/";
"sumo_secret"
string => "ZaVnC4dhaV1-MY_SECRET_KEY";
"curl_args"
string => "-X POST -T $(policy_update_file) $(sumo_url)$(sumo_secret)";
In this next section we tell CFEngine to ensure that the /tmp/CFEngine_policy_updated
file, as defined by the variable policy_update_file
always exists.
We also ensures that the content of this file will be the value of the sys.last_policy_update
variable which we now know is the timestamp. We further set a class called new_policy_update
every time there is a change in the file. This class later becomes the trigger point for when to upload the file to Sumo Logic.
Finally, below you will see a body defining how CFEngine is going to detect changes in policy files, this time using an md5 hash and only looking for change in the content (not permissions or ownership).
files:
"$(policy_update_file)"
create => "true",
edit_line => insert("CFEngine_update: $(sys.last_policy_update)"),
edit_defaults => file;
"$(policy_update_file)"
classes => if_repaired("new_policy_update"),
changes => change_detections;
body changes change_detections
{
hash => "md5";
update_hashes => "true";
report_changes => "content";
report_diffs => "true";
}
The final section in the CFEngine policy is where the command that uploads the file with the timestamp to Sumo Logic is defined.
The command will only be issued whenever a class called new_policy_update
is set, which we above defined to be set when there is a change detection. The handle argument is a useful way to document your intentions.
commands:
new_policy_update::
"/usr/bin/curl"
args => "$(curl_args)",
classes => if_repaired("new_policy_update_sent_to_sumo_logic"),
contain => shell_command,
handle => "New sumo logic event created";
That's it! You can copy and paste the whole policy file at the bottom of this page.
Save the policy file you make as /tmp/sumologic_policy_update.cf
Ensure the policy always runs
Normally, to ensure your policy file is put into action, you would need to follow these two steps:
Move the policy file to your masterfiles directory:
commandmv /tmp/sumo.cf /var/cfengine/masterfiles/
Modify
promises.cf
to include your policyUnless you use version control system, or has a non-standard CFEngine setup, modify your
promises.cf
file by adding the new bundle name and policy-file so it will be picked up by CFEngine and be part of all it future runs.commandvi /var/cfengine/masterfiles/promises.cf
Under the body common control, add sumo_logic_policy_update
to your bundle sequence.
body common control
{
bundlesequence = {
# Common bundle first (Best Practice)
sumo_logic_policy_update,
inventory_control,
...
Under body common control, add /sumologic_policy_update.cf/ to your inputs section.
inputs => {
# File definition for global variables and classes
"sumologic_policy_update.cf",
...
That's all.
Test it!
To test it, we need to make a change to any CFEngine policy, and then go to Sumo Logic to see if there is a new timestamp reported.
- Make a change to any policy file, for examle
promises.cf
:
vi /var/cfengine/masterfiles/promises.cf
Add a comment and close the file.
- Check if timestamp has been updated
cat /tmp/CFEngine_policy_updated
- Check with Sumo Logic
Mission Accomplished!
As we can see above CFEngine detected a change on Thursday Oct 2 at 01:16:42
and also at 01:13:45
.
Source-code:
The policy as found in sumologic_policy_update.cf
.
bundle agent sumo_logic_policy_update
{
vars:
"policy_update_file"
string => "/tmp/CFEngine_policy_updated";
"sumo_url"
string => "https://collectors.sumologic.com/receiver/v1/http/";
"sumo_secret"
string => "MY_SECRET_KEY";
"curl_args"
string => "-X POST -T $(policy_update_file) $(sumo_url)$(sumo_secret)";
files:
"$(policy_update_file)"
create => "true",
edit_line => insert("CFEngine_update: $(sys.last_policy_update)"),
edit_defaults => file;
"$(policy_update_file)"
classes => if_repaired("new_policy_update"),
changes => change_detections;
commands:
new_policy_update::
"/usr/bin/curl"
args => "$(curl_args)",
classes => if_repaired("new_policy_update_sent_to_sumo_logic"),
contain => shell_command,
handle => "New sumo logic event created";
}
body changes change_detections
{
hash => "md5";
update_hashes => "true";
report_changes => "content";
report_diffs => "true";
}
body contain shell_command
{
useshell => "useshell";
}
bundle edit_line insert(str)
{
insert_lines:
"$(str)";
}
body edit_defaults file
{
empty_file_before_editing => "true";
}
Rendering files with Mustache templates
In this tutorial we will show how to use CFEngine to manage file configurations using mustache templating.
How it works
When working with templates, you need a template and data (parameters). Based on this CFEngine will render a file that fills the data in the appropriate places based on the template. Let's create a templating solution for our home grown myapp-application. Below you can see the desired end state of the config file to the left. The right column shows the template we will use.
myapp.conf – desired end state: Port 3508 Protocol 2 Filepath /mypath/ Encryption 256 Loglevel 1 Allowed users thomas=admin malin=guest |
myapp.conf.template – the template:. Port {{port}} Protocol {{protocol}} Filepath {{filepath}} Encryption {{encryption-level}} Loglevel {{loglevel}} Allowed users {{#users}} {{user}}={{level}}{{/users}} |
- Create the template
Create a file called /tmp/myapp.conf.template
with the following content:
Port {{port}}
Protocol {{protocol}}
Filepath {{filepath}}
Encryption {{encryption-level}}
Loglevel {{loglevel}}
Allowed users {{#users}}
{{user}}={{level}}{{/users}}
- Create CFEngine policy
Create a file called /tmp/editconfig.cf
with the following content:
bundle agent myapp_confs
{
files:
"/tmp/myapp.conf"
create => "true",
edit_template => "/tmp/myapp.conf.template",
template_method => "mustache",
template_data => parsejson('
{
"port": 3508,
"protocol": 2,
"filepath": "/mypath/",
"encryption-level": 256,
"loglevel": 1,
"users":
[
{"user": "thomas", "level": "admin"},
{"user": "malin", "level": "guest"}
]
}
');
}
body agent __main__
{
methods:
"myapp_confs";
}
In this policy we tell CFEngine to ensure a file called myapp.conf
exists. The content of the file shall be based on a template file called /tmp/myapp.conf.template
. The template method we use is mustache. Next we define the key value pairs we want to apply using json format (port shall be 3508, protocol 2, etc.)
- Test it out, and verify the result
Run CFEngine:
/var/cfengine/bin/cf-agent /tmp/editconfig.cf
Verify the result:
cat /tmp/myapp.conf
Port 3508
Protocol 2
Filepath /mypath/
Encryption 256
Loglevel 1
Allowed users
thomas=admin
malin=guest
You should now see the existence of a file called /tmp/myapp.conf
with content similar to the desired state described above.
- Congratulation you are done!
With CFEngine you can simplify management of configurations using templating. CFEngine comes both with its own and the mustache templating engine.
PS. If you manually change anything in myapp.conf
, CFEngine will now restore it back to its desired state upon next run.
If there is no change in the template (myapp.conf.template
), the file (myapp.config
), or the data used by the template, CFEngine will not make any changes.
Reporting
No promises made in CFEngine imply automatic aggregation of data to a central location. In CFEngine Enterprise (our commercial version), an optimized aggregation of standardized reports is provided, but the ultimate decision to aggregate must be yours.
Monitoring and reporting capabilities in CFEngine depend on your installation:
Enterprise Edition Reporting
The CFEngine Enterprise edition offers a framework for configuration management that goes beyond building and deploying systems. Features include compliance management, reporting and business integration, and tools for handling the necessary complexity.
In a CFEngine Enterprise installation, the CFEngine Server aggregates information about the environment in a centralized database. By default data is collected every 5 minutes from all bootstrapped hosts and includes information about:
- logs about promises kept, not kept and repaired
- current host contexts and classifications
- variables
- software information
- file changes
This data can be mined using SQL queries and then used for inventory management, compliance reporting, system diagnostics, and capacity planning.
Access to the data is provided through:
Command-Line Reporting
Community Edition
Basic output to file or logs can be customized on a per-promise basis. Users can design their own log and report formats, but data processing and extraction from CFEngine's embedded databases must be scripted by the user.
Note:
If you have regular reporting needs, we recommend using our commercially-supported version of CFEngine, Enterprise. It will save considerable time and resources in programming, and you will have access to the latest developments through the software subscription.
Monitoring and reporting
What are monitoring and reporting?
Monitoring is the sampling of system variables at regular intervals in order to present an overview of actual changes taking place over time. Monitoring data are often presented as extensive views of moving-line time series. Monitoring has the ability to detect anomalous behavior by comparing past and present.
The term reporting is usually taken to mean the creation of short summaries of specific system properties suitable for management. System reports describe both promises about the system, such as compliance, discovered changes and faults.
The challenge of both these activities is to compare intended or promised, behavior with the actual observed behavior of the system.
Should monitoring and configuration be separate?
The traditional view of IT operations is that configuration, monitoring, and reporting are three different things that should not be joined. Traditionally, all three have been independent centralized processes. This view has emerged historically, but it has a major problem: Humans are needed to glue these parts back together. Monitoring as an independent activity is inherently non-scalable. When numbers of hosts grow beyond a few thousands, centralized monitoring schemes fail to manage the information. Tying configuration (and therefore repair) to monitoring at the host level is essential for the effective management of large and distributed data facilities. CFEngine foresaw this need in 1998, with its Computer Immunology initiative, and continues to develop this strategy.
CFEngine's approach is to focus on scalability. The commercial editions of CFEngine provide what meaningful information they can in a manner that can be scaled to tens of thousands of machines.
Command-Line reports
Command-line reporting is available to Enterprise and Community users.
Overview
The following report topics are included:
CFEngine output levels
CFEngine's default behavior is to report to the console (known as standard output). It's default behavior is to report nothing except errors that are judged to be of a critical nature.
By using CFEngine with the inform flag, you can alter the default to report on action items (actual changes) and warnings:
# cf-agent -I
# cf-agent --inform
By using CFEngine with the verbose flag, you can alter the default to report all of its thought-processes. You should not interpret a message that only appears in CFEngine's verbose mode as an actual error, only as information that might be relevant to decisions being made by the agent:
# cf-agent -v
# cf-agent --verbose
Creating custom reports
CFEngine allows you to use reports
promises to make reports of your own. A simple
example of this is shown below.
body common control
{
bundlesequence => { "test" };
}
#
bundle agent test
{
reports:
cfengine_3::
"$(sys.date),This is a report"
report_to_file => "/tmp/test_log";
}
We can apply this idea to make more useful custom reports. In this example, the agent tests for certain software package and creates a simple HTML file of existing software:
body common control
{
bundlesequence => { "test" };
}
#
bundle agent test
{
vars:
"software" slist => { "gpg", "zip", "rsync" };
classes:
"no_report" expression => fileexists("/tmp/report.html");
"have_$(software)" expression => fileexists("/usr/bin/$(software)");
reports:
no_report::
"
<html>
Name of this host is: $(sys.host)<br>
Type of this host is: $(sys.os)<br>
"
report_to_file => "/tmp/report.html";
#
"
Host has software $(software)<br>
"
if => "have_$(software)",
report_to_file => "/tmp/report.html";
#
"
</html>
"
report_to_file => "/tmp/report.html";
}
The outcome of this promise is a file called /tmp/report.html which contains the following output:
<html>
Name of this host is: atlas<br>
Type of this host is: linux<br>
Host has software gpg<br>
Host has software zip<br>
Host has software rsync<br>
</html>
The mechanism shown above can clearly be used to create a wide variety of report formats, but it requires a lot of coding and maintenance by the user.
Including data in reports
CFEngine generates information internally that you might want to use in reports.
For example, the agent cf-agent
interfaces with the local light-weight monitoring agent
cf-monitord
so that system state can be reported simply:
body common control
{
bundlesequence => { "report" };
}
###########################################################
bundle agent report
{
reports:
linux::
"/etc/passwd except $(const.n)"
showstate => { "otherprocs", "rootprocs" };
}
A bonus to this is that you can get CFEngine to report system anomalies:
reports:
rootprocs_high_dev2::
"RootProc anomaly high 2 dev on $(mon.host) at approx $(mon.env_time)
measured value $(mon.value_rootprocs)
average $(mon.average_rootprocs) pm $(mon.stddev_rootprocs)"
showstate => { "rootprocs" };
entropy_www_in_high&anomaly_hosts.www_in_high_anomaly::
"High entropy incoming www anomaly on $(mon.host) at $(mon.env_time)
measured value $(mon.value_www_in)
average $(mon.average_www_in) pm $(mon.stddev_www_in)"
showstate => { "incoming.www" };
This produces the following standard output:
R: State of otherprocs peaked at Tue Dec 1 12:12:21 2014
R: The peak measured state was q = 98:
R: Frequency: [kjournald] |** (2/98)
R: Frequency: [pdflush] |** (2/98)
R: Frequency: /var/cfengine/bin/cf-execd|** (2/98)
R: Frequency: COMMAND |* (1/98)
R: Frequency: init [5] |* (1/98)
R: Frequency: [kthreadd] |* (1/98)
R: Frequency: [migration/0] |* (1/98)
R: Frequency: [ksoftirqd/0] |* (1/98)
R: Frequency: [events/0] |* (1/98)
R: Frequency: [khelper] |* (1/98)
R: Frequency: [kintegrityd/0] |* (1/98)
Finally, you can quote lines from files in your data for convenience:
body common control
{
bundlesequence => { "report" };
}
###########################################################
bundle agent report
{
reports:
linux::
"/etc/passwd except $(const.n)"
printfile => pr("/etc/passwd","5");
}
######################################################################
body printfile pr(file,lines)
{
file_to_print => "$(file)";
number_of_lines => "$(lines)";
}
This produces the following output:
R: /etc/passwd except
R: at:x:25:25:Batch jobs daemon:/var/spool/atjobs:/bin/bash
R: avahi:x:103:105:User for Avahi:/var/run/avahi-daemon:/bin/false
R: beagleindex:x:104:106:User for Beagle indexing:/var/cache/beagle:/bin/bash
R: bin:x:1:1:bin:/bin:/bin/bash
R: daemon:x:2:2:Daemon:/sbin:/bin/bash
Excluding data from reports
CFEngine generates information internally that you might want exclude from
reports. Any promise outcome can be excluded from report collection based on its
handle. vars
and classes
type promises can be excluded using its handle
or by meta tag.
bundle agent main
{
files:
linux::
"/var/log/noisy.log"
handle => "noreport_noisy_log_rotation",
rename => rotate(5);
}
body report_data_select default_data_select_policy_hub
# @brief Data to collect from policy servers by default
#
# By convention variables and classes known to be internal, (having no
# reporting value) should be prefixed with an underscore. By default the policy
# framework explicitly excludes these variables and classes from collection.
{
# Collect all classes or vars tagged with `inventory` or `report`
metatags_include => { "inventory", "report" };
# Exclude any classes or vars tagged with `noreport`
metatags_exclude => { "noreport" };
# Exclude any promise with handle matching `noreport_.*` from report collection.
promise_handle_exclude => { "noreport_.*" };
# Include all metrics from cf-monitord
monitoring_include => { ".*" };
}
Creating custom logs
Logs can be attached to any promise. In this example, an executed shell command logs a message to the standard output. CFEngine recognizes thestdoutfilename for Standard Output, in the Unix/C standard manner:
bundle agent test
{
commands:
"/tmp/myjob",
action => logme("executor");
}
############################################
body action logme(x)
{
log_repaired => "stdout";
log_string => " -> Started the $(x) (success)";
}
In the following example, a file creation promise logs different outcomes (success or failure) to different log files:
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
vars:
"software" slist => { "/root/xyz", "/tmp/xyz" };
files:
"$(software)"
create => "true",
action => logme("$(software)");
}
#
body action logme(x)
{
log_kept => "/tmp/private_keptlog.log";
log_failed => "/tmp/private_faillog.log";
log_repaired => "/tmp/private_replog.log";
log_string => "$(sys.date) $(x) promise status";
}
This generates three different logs with the following output:
more /tmp/private_keptlog.log
Sun Dec 6 11:58:16 2009 /tmp/xyz promise status
Sun Dec 6 11:58:43 2009 /tmp/xyz promise status
Redirecting output to logs
CFEngine interfaces with the system logging tools in different ways. Syslog
is the
default log for Unix-like systems, while the event logger
is the default on Windows.
You may choose to copy a fixed level of CFEngine's standard screen messaging to the
system logger on a per-promise basis:
body common control
{
bundlesequence => { "one" };
}
bundle agent one
{
files:
"/tmp/xyz"
create => "true",
action => log;
}
body action log
{
log_level => "inform";
}
Change detection: tripwires
Doing a change detection scan is a convergent process, but it can still detect changes and present the data in a compressed format that is often more convenient than a full-scale audit. The result is less precise, but there is a trade-off between precision and cost.
To make a change tripwire, use a files promise, as shown below:
body common control
{
bundlesequence => { "testbundle" };
}
#
bundle agent testbundle
{
files:
"/home/mark/tmp" -> "me"
changes => scan_files,
depth_search => recurse("inf");
}
# library code ...
body changes scan_files
{
report_changes => "all";
update_hashes => "true";
}
body depth_search recurse(d)
{
depth => "$(d)";
}
In CFEngine Enterprise, reports of the following form are generated when these promises are kept by the agent:
Change detected File change
Sat Dec 5 18:27:44 2013 group for /tmp/testfile changed 100 -> 0
Sat Dec 5 18:27:44 2013 /tmp/testfile
Sat Dec 5 18:20:45 2013 /tmp/testfile
These reports are generated automatically in Enterprise, and are integrated into the web-browsable knowledge map. Community edition users must extract the data and create these themselves.
File comparison
- Add the policy contents (also can be downloaded from file_compare_test.cf) to a new file, such as /var/cfengine/masterfiles/file_test.cf.
Run the following commands as root on the command line: ```console export AOUT_BIN="a.out" export GCC_BIN="/usr/bin/gcc" export RM_BIN="/bin/rm" export WORK_DIR=$HOME export CFE_FILE1="test_plain_1.txt" export CFE_FILE2="test_plain_2.txt"
/var/cfengine/bin/cf-agent /var/cfengine/masterfiles/file_test.cf --bundlesequence robot,global_vars,packages,create_aout_source_file,create_aout,test_delete,do_files_exist_1,create_file_1,outer_bundle_1,copy_a_file,do_files_exist_2,list_file_1,stat,outer_bundle_2,list_file_2 ```
Here is the order in which bundles are called in the command line above (some other support bundles are contained within file_test.cf but are not included here):
- robot - demonstrates use of
reports
. - global_vars - sets up some global variables for later use.
- packages - installs packages that will be used later on.
- create_aout_source_file - creates a source file.
- create_aout - compiles the source file.
- test_delete - deletes a file.
- do_files_exist_1 - checks the existence of files.
- create_file_1 - creates a file.
- outer_bundle_1 - adds text to a file.
- copy_a_file - copies the file.
- do_files_exist_2 - checks the existence of both files.
- list_file_1 - shows the contents of each file.
- stat - uses the stat command and the aout application to compare modified times of both files.
- outer_bundle_2 - modifies the contents of the second file.
- list_file_2 - shows the contents of both files and uses CFEngine functionality to compare the modified time for each file.
robot
Demonstrates use of reports
, using an ascii art representation of the CFEngine robot.
global_vars
Sets up some global variables that are used frequently by other bundles.
bundle common global_vars
{
vars:
"gccexec" string => getenv("GCC_BIN",255);
"rmexec" string => getenv("RM_BIN",255);
"aoutbin" string => getenv("AOUT_BIN",255);
"workdir" string => getenv("WORK_DIR",255);
"aoutexec" string => "$(workdir)/$(aoutbin)";
"file1name" string => getenv("CFE_FILE1",255);
"file2name" string => getenv("CFE_FILE2",255);
"file1" string => "$(workdir)/$(file1name)";
"file2" string => "$(workdir)/$(file2name)";
classes:
"gclass" expression => "any";
}
packages
Ensures that the gcc package is installed, for later use by the create_aout bundle.
bundle agent packages
{
vars:
"match_package"
slist => {
"gcc"
};
packages:
"$(match_package)"
package_policy => "add",
package_method => yum;
reports:
gclass::
"Package gcc installed";
"*********************************";
}
create_aout_source_file
Creates the c source file that will generate a binary application in create_aout.
bundle agent create_aout_source_file
{
# This bundle creates the source file that will be compiled in bundle agent create_aout.
# See that bunlde's comments for more information.
vars:
# An slist is used here instead of a straight forward string because it doesn't seem possible to create
# line endings using \n when using a string to insert text into a file.
"c"
slist => {
"#include <stdlib.h>",
"#include <stdio.h>",
"#include <sys/stat.h>",
"#include <string.h>",
"void main()",
"{char file1[255];strcpy(file1,\"$(global_vars.file1)\");char file2[255];strcpy(file2,\"$(global_vars.file2)\");struct stat time1;int i = lstat(file1, &time1);struct stat time2;int j = lstat(file2, &time2);if (time1.st_mtime < time2.st_mtime){printf(\"Newer\");}else{if(time1.st_mtim.tv_nsec < time2.st_mtim.tv_nsec){printf(\"Newer\");}else{printf(\"Not newer\");}}}"
};
files:
"$(global_vars.workdir)/a.c"
perms => system,
create => "true",
edit_line => Insert("@(c)");
reports:
"The source file $(global_vars.workdir)/a.c has been created. It will be used to compile the binary a.out, which will provide more accurate file stats to compare two files than the built in CFEngine functionality for comparing file stats, including modification time. This information will be used to determine of the second of the two files being compared is newer or not.";
"*********************************";
}
create_aout
This bundle creates a binary application from the source in create_aout_source_file that uses the stat library to compare two files, determine if the modified times are different, nd whether the second file is newer than the first.
The difference between this application and using CFEngine's built in support for getting file stats is that normally the accuracy is only to the second of the modified file time but in order to better compare two files requires parts of a second as well. The stat library provides the extra support for retrieving the additional information required.
bundle agent create_aout
{
classes:
"doesfileacexist" expression => fileexists("$(global_vars.workdir)/a.c");
"doesaoutexist" expression => fileexists("$(global_vars.aoutbin)");
vars:
# Removes any previous binary
"rmaout" string => execresult("$(global_vars.rmexec) $(global_vars.aoutexec)","noshell");
doesfileacexist::
"compilestr" string => "$(global_vars.gccexec) $(global_vars.workdir)/a.c -o $(global_vars.aoutexec)";
"gccaout" string => execresult("$(compilestr)","noshell");
reports:
doesfileacexist::
"gcc output: $(gccaout)";
"Creating aout using $(compilestr)";
!doesfileacexist::
"Cannot compile a.out, $(global_vars.workdir)/a.c does not exist.";
doesaoutexist::
"The binary application aout has been compiled from the source in the create_aout_source_file bundle. It uses the stat library to compare two files, determine if the modified times are different, and whether the second file is newer than the first. The difference between this application and using CFEngine's built in support for getting file stats (e.g. filestat, isnewerthan), which provides file modification time accurate to a second. However, in order to better compare two files might sometimes require parts of a second as well. The stat library provides the extra support for retrieving the additional information required to get better accuracy (down to parts of a second), and is utilized by the binary application a.out that is compiled within the create_aout bundle.";
"*********************************";
}
test_delete
Deletes any previous copy of the test files used in the example.
bundle agent test_delete
{
files:
"$(global_vars.file1)"
delete => tidy;
}
do_files_exist_1
Verifies whether the test files exist or not.
bundle agent do_files_exist_1
{
classes:
"doesfile1exist"
expression => fileexists("$(global_vars.file1)");
"doesfile2exist"
expression => fileexists("$(global_vars.file2)");
methods:
doesfile1exist::
"any" usebundle => delete_file("$(global_vars.file1)");
doesfile2exist::
"any" usebundle => delete_file("$(global_vars.file2)");
reports:
!doesfile1exist::
"$(global_vars.file1) does not exist.";
doesfile1exist::
"$(global_vars.file1) did exist. Call to delete it was made.";
!doesfile2exist::
"$(global_vars.file2) does not exist.";
doesfile2exist::
"$(global_vars.file2) did exist. Call to delete it was made.";
}
create_file_1
Creates the first test file, as an empty file.
bundle agent create_file_1
{
files:
"$(global_vars.file1)"
perms => system,
create => "true";
reports:
"$(global_vars.file1) has been created";
}
outer_bundle_1
Adds some text to the first test file.
bundle agent outer_bundle_1
{
files:
"$(global_vars.file1)"
create => "false",
edit_line => inner_bundle_1;
}
copy_a_file
Makes a copy of the test file.
bundle agent copy_a_file
{
files:
"$(global_vars.file2)"
copy_from => local_cp("$(global_vars.file1)");
reports:
"$(global_vars.file1) has been copied to $(global_vars.file2)";
}
do_files_exist_2
Verifies that both test files exist.
bundle agent do_files_exist_2
{
methods:
"any" usebundle => does_file_exist($(global_vars.file1));
"any" usebundle => does_file_exist($(global_vars.file2));
}
list_file_1
Reports the contents of each test file.
bundle agent list_file_1
{
methods:
"any" usebundle => file_content($(global_vars.file1));
"any" usebundle => file_content($(global_vars.file2));
reports:
"*********************************";
}
exec_aout
bundle agent exec_aout
{
classes:
"doesaoutexist"
expression => fileexists("$(global_vars.aoutbin)");
vars:
doesaoutexist::
"aout"
string => execresult("$(global_vars.aoutexec)","noshell");
reports:
doesaoutexist::
"*********************************";
"$(global_vars.aoutbin) determined that $(global_vars.file2) is $(aout) than $(global_vars.file1)";
"*********************************";
!doesaoutexist::
"Executable $(global_vars.aoutbin) does not exist.";
}
stat
Compares the modified time of each test file using the binary application compiled in create_aout to see if it is newer.
bundle agent stat
{
classes:
"doesfile1exist"
expression => fileexists("$(global_vars.file1)");
"doesfile2exist"
expression => fileexists("$(global_vars.file2)");
vars:
doesfile1exist::
"file1" string => "$(global_vars.file1)";
"file2" string => "$(global_vars.file2)";
"file1_stat" string => execresult("/usr/bin/stat -c \"%y\" $(file1)","noshell");
"file1_split1" slist => string_split($(file1_stat)," ",3);
"file1_split2" string => nth("file1_split1",1);
"file1_split3" slist => string_split($(file1_split2),"\.",3);
"file1_split4" string => nth("file1_split3",1);
"file2_stat" string => execresult("/usr/bin/stat -c \"%y\" $(file2)","noshell");
"file2_split1" slist => string_split($(file2_stat)," ",3);
"file2_split2" string => nth("file2_split1",1);
"file2_split3" slist => string_split($(file2_split2),"\.",3);
"file2_split4" string => nth("file2_split3",1);
methods:
"any" usebundle => exec_aout();
reports:
doesfile1exist::
"Parts of a second extracted extracted from stat for $(file1): $(file1_split4). Full stat output for $(file1): $(file1_stat)";
"Parts of a second extracted extracted from stat for $(file2): $(file2_split4). Full stat output for $(file2): $(file2_stat)";
"Using the binary Linux application stat to compare two files can help determine if the modified times between two files are different. The difference between the stat application using its additional flags and using CFEngine's built in support for getting and comparing file stats (e.g. filestat, isnewerthan) is that normally the accuracy is only to the second of the file's modified time. In order to better compare two files requires parts of a second as well, which the stat command can provide with some additional flags. Unfortunately the information must be extracted from the middle of a string, which is what the stat bundle accomplishes using the string_split and nth functions.";
"*********************************";
!doesfile1exist::
"stat: $(global_vars.file1) and probably $(global_vars.file2) do not exist.";
}
outer_bundle_2
Modifies the text in the second file.
bundle agent outer_bundle_2
{
files:
"$(global_vars.file2)"
create => "false",
edit_line => inner_bundle_2;
}
list_file_2
Uses filestat
and isnewerthan
to compare the two test files to see if the second one is newer. Sometimes the modifications already performed, such as copy and modifying text, happen too quickly and filestat and isnewerthan may both report that the second test file is not newer than the first, while the more accurate stat based checks in the stat bundle (see step 12) will recognize the difference.
bundle agent list_file_2
{
methods:
"any" usebundle => file_content($(global_vars.file1));
"any" usebundle => file_content($(global_vars.file2));
classes:
"ok" expression => isgreaterthan(filestat("$(global_vars.file2)","mtime"),filestat("$(global_vars.file1)","mtime"));
"newer" expression => isnewerthan("$(global_vars.file2)","$(global_vars.file1)");
reports:
"*********************************";
ok::
"Using isgreaterthan+filestat determined that $(global_vars.file2) was modified later than $(global_vars.file1).";
!ok::
"Using isgreaterthan+filestat determined that $(global_vars.file2) was not modified later than $(global_vars.file1).";
newer::
"Using isnewerthan determined that $(global_vars.file2) was modified later than $(global_vars.file1).";
!newer::
"Using isnewerthan determined that $(global_vars.file2) was not modified later than $(global_vars.file1).";
}
Full policy
body common control {
inputs => {
"libraries/cfengine_stdlib.cf",
};
}
bundle agent robot
{
reports:
" 77777777777";
" 77777777777777";
" 777 7777 777";
" 7777777777777";
" 777777777777";
" 777 7777 77";
" ";
" ZZZZ ZZZ ZZZZ ZZZ ZZZZ";
" ZZZZZ ZZZZZZZZZZZZZZ ZZZZZ ";
" ZZZZZZZ ZZZZZZZZZZZZZ ZZZZZZZ";
" ZZZZ ------------- ZZZZZZ";
" ZZZZZ !CFENGINE! ZZZZZ";
" ZZZZ ------------- ZZZZZ";
" ZZZZZ ZZZZZZZZZZZZZZ ZZZZZ";
" ZZZ ZZZZZZZZZZZZZ ZZZ";
" ZZZZZ ZZZZZZZZZZZZZ ZZZZZ";
" ..?ZZZ+,,,,, ZZZZZZZZZZZZZZ ZZZZZ";
" ...ZZZZ~ ,:: ZZZZZZZZZZZZZ ZZZZ";
" ..,ZZZZZ,:::::: ZZZZZ";
" ZZZ ZZZ";
" ~ ===+";
" ZZZZZZZZZZZZZI??";
" ZZZZZZZZZZZZZ$???";
" 7Z$+ ZZ ZZZ???II";
" ZZZZZ+ ZZZZZIIII";
" ZZZZZ ZZZZZ III77";
" +++ +$ZZ??? ZZZ";
" +++??ZZZZZIIIIZZZZZ";
" ????ZZZZZIIIIZZZZZ";
" ??IIZZZZ 7777ZZZ";
" IIZZZZZ 77ZZZZZ";
" I$ZZZZ $ZZZZ";
}
bundle common global_vars
{
vars:
"gccexec" string => getenv("GCC_BIN",255);
"rmexec" string => getenv("RM_BIN",255);
"aoutbin" string => getenv("AOUT_BIN",255);
"workdir" string => getenv("WORK_DIR",255);
"aoutexec" string => "$(workdir)/$(aoutbin)";
"file1name" string => getenv("CFE_FILE1",255);
"file2name" string => getenv("CFE_FILE2",255);
"file1" string => "$(workdir)/$(file1name)";
"file2" string => "$(workdir)/$(file2name)";
classes:
"gclass" expression => "any";
}
bundle agent packages
{
vars:
"match_package" slist => {
"gcc"
};
packages:
"$(match_package)"
package_policy => "add",
package_method => yum;
reports:
gclass::
"Package gcc installed";
"*********************************";
}
bundle agent create_aout_source_file
{
# This bundle creates the source file that will be compiled in bundle agent create_aout.
# See that bunlde's comments for more information.
vars:
# An slist is used here instead of a straight forward string because it doesn't seem possible to create
# line endings using \n when using a string to insert text into a file.
"c" slist => {"#include <stdlib.h>","#include <stdio.h>","#include <sys/stat.h>","#include <string.h>","void main()","{char file1[255];strcpy(file1,\"$(global_vars.file1)\");char file2[255];strcpy(file2,\"$(global_vars.file2)\");struct stat time1;int i = lstat(file1, &time1);struct stat time2;int j = lstat(file2, &time2);if (time1.st_mtime < time2.st_mtime){printf(\"Newer\");}else{if(time1.st_mtim.tv_nsec < time2.st_mtim.tv_nsec){printf(\"Newer\");}else{printf(\"Not newer\");}}}"};
files:
"$(global_vars.workdir)/a.c"
perms => system,
create => "true",
edit_line => Insert("@(c)");
reports:
"The source file $(global_vars.workdir)/a.c has been created. It will be used to compile the binary a.out, which will provide more accurate file stats to compare two files than the built in CFEngine functionality for comparing file stats, including modification time. This information will be used to determine of the second of the two files being compared is newer or not.";
"*********************************";
}
bundle edit_line Insert(name)
{
insert_lines:
"$(name)";
}
bundle agent create_aout
{
classes:
"doesfileacexist" expression => fileexists("$(global_vars.workdir)/a.c");
"doesaoutexist" expression => fileexists("$(global_vars.aoutbin)");
vars:
# Removes any previous binary
"rmaout" string => execresult("$(global_vars.rmexec) $(global_vars.aoutexec)","noshell");
doesfileacexist::
"compilestr" string => "$(global_vars.gccexec) $(global_vars.workdir)/a.c -o $(global_vars.aoutexec)";
"gccaout" string => execresult("$(compilestr)","noshell");
reports:
doesfileacexist::
"gcc output: $(gccaout)";
"Creating aout using $(compilestr)";
!doesfileacexist::
"Cannot compile a.out, $(global_vars.workdir)/a.c does not exist.";
doesaoutexist::
"The binary application aout has been compiled from the source in the create_aout_source_file bundle. It uses the stat library to compare two files, determine if the modified times are different, and whether the second file is newer than the first. The difference between this application and using CFEngine's built in support for getting file stats (e.g. filestat, isnewerthan), which provides file modification time accurate to a second. However, in order to better compare two files might sometimes require parts of a second as well. The stat library provides the extra support for retrieving the additional information required to get better accuracy (down to parts of a second), and is utilized by the binary application a.out that is compiled within the create_aout bundle.";
"*********************************";
}
bundle agent test_delete
{
files:
"$(global_vars.file1)"
delete => tidy;
}
bundle agent delete_file(fname)
{
files:
"$(fname)"
delete => tidy;
reports:
"Deleted $(fname)";
}
body contain del_file
{
useshell => "useshell";
}
bundle agent do_files_exist_1
{
classes:
"doesfile1exist" expression => fileexists("$(global_vars.file1)");
"doesfile2exist" expression => fileexists("$(global_vars.file2)");
methods:
doesfile1exist::
"any" usebundle => delete_file("$(global_vars.file1)");
doesfile2exist::
"any" usebundle => delete_file("$(global_vars.file2)");
reports:
!doesfile1exist::
"$(global_vars.file1) does not exist.";
doesfile1exist::
"$(global_vars.file1) did exist. Call to delete it was made.";
!doesfile2exist::
"$(global_vars.file2) does not exist.";
doesfile2exist::
"$(global_vars.file2) did exist. Call to delete it was made.";
}
bundle agent create_file_1
{
files:
"$(global_vars.file1)"
perms => system,
create => "true";
reports:
"$(global_vars.file1) has been created";
}
bundle agent outer_bundle_1
{
files:
"$(global_vars.file1)"
create => "false",
edit_line => inner_bundle_1;
}
bundle agent copy_a_file
{
files:
"$(global_vars.file2)"
copy_from => local_cp("$(global_vars.file1)");
reports:
"$(global_vars.file1) has been copied to $(global_vars.file2)";
"*********************************";
}
bundle agent do_files_exist_2
{
methods:
"any" usebundle => does_file_exist($(global_vars.file1));
"any" usebundle => does_file_exist($(global_vars.file2));
}
bundle agent does_file_exist(filename)
{
vars:
"filestat" string => filestat("$(filename)","mtime");
classes:
"fileexists" expression => fileexists("$(filename)");
reports:
fileexists::
"$(filename) exists. Last Modified Time = $(filestat).";
!fileexists::
"$(filename) does not exist";
}
bundle agent list_file_1
{
methods:
"any" usebundle => file_content($(global_vars.file1));
"any" usebundle => file_content($(global_vars.file2));
reports:
"*********************************";
}
bundle agent exec_aout
{
classes:
"doesaoutexist" expression => fileexists("$(global_vars.aoutbin)");
vars:
doesaoutexist::
"aout" string => execresult("$(global_vars.aoutexec)","noshell");
reports:
doesaoutexist::
"*********************************";
"$(global_vars.aoutbin) determined that $(global_vars.file2) is $(aout) than $(global_vars.file1)";
"*********************************";
!doesaoutexist::
"Executable $(global_vars.aoutbin) does not exist.";
}
bundle agent stat
{
classes:
"doesfile1exist" expression => fileexists("$(global_vars.file1)");
"doesfile2exist" expression => fileexists("$(global_vars.file2)");
vars:
doesfile1exist::
"file1" string => "$(global_vars.file1)";
"file2" string => "$(global_vars.file2)";
"file1_stat" string => execresult("/usr/bin/stat -c \"%y\" $(file1)","noshell");
"file1_split1" slist => string_split($(file1_stat)," ",3);
"file1_split2" string => nth("file1_split1",1);
"file1_split3" slist => string_split($(file1_split2),"\.",3);
"file1_split4" string => nth("file1_split3",1);
"file2_stat" string => execresult("/usr/bin/stat -c \"%y\" $(file2)","noshell");
"file2_split1" slist => string_split($(file2_stat)," ",3);
"file2_split2" string => nth("file2_split1",1);
"file2_split3" slist => string_split($(file2_split2),"\.",3);
"file2_split4" string => nth("file2_split3",1);
methods:
"any" usebundle => exec_aout();
reports:
doesfile1exist::
"Parts of a second extracted extracted from stat for $(file1): $(file1_split4). Full stat output for $(file1): $(file1_stat)";
"Parts of a second extracted extracted from stat for $(file2): $(file2_split4). Full stat output for $(file2): $(file2_stat)";
"Using the binary Linux application stat to compare two files can help determine if the modified times between two files are different. The difference between the stat application using its additional flags and using CFEngine's built in support for getting and comparing file stats (e.g. filestat, isnewerthan) is that normally the accuracy is only to the second of the file's modified time. In order to better compare two files requires parts of a second as well, which the stat command can provide with some additional flags. Unfortunately the information must be extracted from the middle of a string, which is what the stat bundle accomplishes using the string_split and nth functions.";
"*********************************";
!doesfile1exist::
"stat: $(global_vars.file1) and probably $(global_vars.file2) do not exist.";
}
bundle agent outer_bundle_2
{
files:
"$(global_vars.file2)"
create => "false",
edit_line => inner_bundle_2;
}
bundle edit_line inner_bundle_1
{
vars:
"msg" string => "Helloz to World!";
insert_lines:
"$(msg)";
reports:
"inserted $(msg) into $(global_vars.file1)";
}
bundle edit_line inner_bundle_2
{
replace_patterns:
"Helloz to World!"
replace_with => hello_world;
reports:
"Text in $(global_vars.file2) has been replaced";
}
body replace_with hello_world
{
replace_value => "Hello World";
occurrences => "all";
}
bundle agent list_file_2
{
methods:
"any" usebundle => file_content($(global_vars.file1));
"any" usebundle => file_content($(global_vars.file2));
classes:
"ok" expression => isgreaterthan(filestat("$(global_vars.file2)","mtime"),filestat("$(global_vars.file1)","mtime"));
"newer" expression => isnewerthan("$(global_vars.file2)","$(global_vars.file1)");
reports:
"*********************************";
ok::
"Using isgreaterthan+filestat determined that $(global_vars.file2) was modified later than $(global_vars.file1).";
!ok::
"Using isgreaterthan+filestat determined that $(global_vars.file2) was not modified later than $(global_vars.file1).";
newer::
"Using isnewerthan determined that $(global_vars.file2) was modified later than $(global_vars.file1).";
!newer::
"Using isnewerthan determined that $(global_vars.file2) was not modified later than $(global_vars.file1).";
}
bundle agent file_content(filename)
{
vars:
"file_content" string => readfile( "$(filename)" , "0" );
"file_stat" string => filestat("$(filename)","mtime");
reports:
"Contents of $(filename) = $(file_content). Last Modified Time = $(file_stat).";
#"The report on contents will only show new content and modifications. Even if the method is called more than once, if the evaluation is exactly the same as the previous call then there will be no report (possibly because the bundle is not evaluated a second time?).";
}
body perms system
{
mode => "0640";
}
Writing and serving policy
About policies and promises
Central to CFEngine's effectiveness in system administration is the concept of a "promise," which defines the intent and expectation of how some part of an overall system should behave.
CFEngine emphasizes the promises a client makes to the overall CFEngine network. Combining promises with patterns to describe where and when promises should apply is what CFEngine is all about.
This document describes in brief what a promise is and what a promise does. There are other resources for finding out additional details about "promises" in the See also section at the end of this document.
What are promises
A promise is the documentation or definition of an intention to act or behave in some manner. They are the rules which CFEngine clients are responsible for implementing.
The value of a promise
When you make a promise it is an effort to improve trust, which is an economic time-saver. If you have trust then there is less need to verify, which in turn saves time and money.
When individual components are empowered with clear guidance, independent decision making power, and the trust that they will fulfil their duties, then systems that are complex and scalable, yet still manageable, become possible.
Anatomy of a promise
bundle agent hello_world
{
reports:
any::
"Hello World!"
comment => "This is a simple promise saying hello to the world.";
}
How promises work
Everything in CFEngine can be thought of as a promise to be kept by different resources in the system. In a system that delivers a web site using Apache, an important promise may be to make sure that the httpd
or apache
package is installed, running, and accessible on port 80.
Summary for writing, deploying and using promises
Writing, deploying, and using CFEngine promises
will generally follow these simple steps:
- Using a text editor, create a new file (e.g.
hello_world.cf
). - Create a bundle and promise in the file (see "Hello world" policy example).
- Save the file on the policy server somewhere under
/var/cfengine/masterfiles
(can be under a sub-directory). - Let CFEngine know about the
promise
on thepolicy server
, generally in the file/var/cfengine/masterfiles/promises.cf
, or a file elsewhere but referred to inpromises.cf
.
* Optional: it is also possible to call a bundle manually, using `cf-agent`.
- Verify the
policy file
was deployed and successfully run.
See Tutorial for running examples for a more detailed step by step tutorial.
Policy workflow
CFEngine does not make absolute choices for you, like other tools. Almost everything about its behavior is a matter of policy and can be changed.
In order to keep operations as simple as possible, CFEngine maintains a private
working directory on each machine, referred to in documentation as WORKDIR
and
in policy by the variable sys.workdir
By default, this is located at
/var/cfengine
or C:\var\CFEngine
. It contains everything CFEngine needs to
run.
The figure below shows how decisions flow through the parts of a system.
It makes sense to have a single point of coordination. Decisions are therefore usually made in a single location (the Policy Definition Point). The history of decisions and changes can be tracked by a version control system of your choice (e.g. Git, Subversion, CVS etc.).
Decisions are made by editing CFEngine's policy file
promises.cf
(or one of its included sub-files). This process is carried out off-line.Once decisions have been formalized and coded, this new policy is copied to a decision distribution point,
sys.masterdir
which defaults to/var/cfengine/masterfiles
on all policy distribution servers.Every client machine contacts the policy server and downloads these updates. The policy server can be replicated if the number of clients is very large, but we shall assume here that there is only one policy server.
Once a client machine has a copy of the policy, it extracts only those promise proposals that are relevant to it, and implements any changes without human assistance. This is how CFEngine manages change.
CFEngine tries to minimize dependencies by decoupling processes. By following this pull-based architecture, CFEngine will tolerate network outages and will recover from deployment errors easily. By placing the burden of responsibility for decision at the top, and for implementation at the bottom, we avoid needless fragility and keep two independent quality assurance processes apart.
Best practices
Policy style guide This covers punctuation, whitespace, and other styles to remember when writing policy.
Bundles best practices Refer to this page as you decide when to make a bundle and when to use classes and/or variables in them.
Testing policies This page describes how to locally test CFEngine and play with configuration files.
See also
Layers of abstraction in policy
CFEngine offers a number of layers of abstraction. The most fundamental atom in CFEngine is the promise. Promises can be made about many system issues, and you described in what context promises are to be kept.
CFEngine is designed to handle high level simplicity (without sacrificing low level capability) by working with configuration patterns. After all, configuration is all about promising consistent patterns in the resources of the system. Lists, for instance, are a particularly common kind of pattern: for each of the following... make a similar promise. There are several ways to organize patterns, using containers, lists and associative arrays.
Menu level
At this high level, a user selects
from a set of pre-defined services
(or
bundles in CFEngine parlance). The selection is not made by every host, rather
one places hosts into roles that will keep certain promises.
bundle agent service_catalogue # menu
{
methods:
any:: # selected by everyone
"everyone" usebundle => time_management,
comment => "Ensure clocks are synchronized";
"everyone" usebundle => garbage_collection,
comment => "Clear junk and rotate logs";
mailservers:: # selected by hosts in class
"mail server" -> { "goal_3", "goal_1", "goal_2" }
usebundle => app_mail_postfix,
comment => "The mail delivery agent";
"mail server" -> goal_3,
usebundle => app_mail_imap,
comment => "The mail reading service";
"mail server" -> goal_3,
usebundle => app_mail_mailman,
comment => "The mailing list handler";
}
Bundle level
At this level, users can switch on and off predefined features, or re-use standard methods, e.g. for editing files:
body common control
{
bundlesequence => {
webserver("on"),
dns("on"),
security_set("on"),
ftp("off")
};
}
The set of bundles that can be selected from is extensible by the user.
Promise level
This is the most detailed level of configuration, and gives full convergent promise behavior to the user. At this promise level, you can specify every detail of promise-keeping behavior, and combine promises together, reusing bundles and methods from standard libraries, or creating your own.
bundle agent addpasswd
{
vars:
# want to set these values by the names of their array keys
"pwd[mark]" string => "mark:x:1000:100:Mark B:/home/mark:/bin/bash";
"pwd[fred]" string => "fred:x:1001:100:Right Said:/home/fred:/bin/bash";
"pwd[jane]" string => "jane:x:1002:100:Jane Doe:/home/jane:/bin/bash";
files:
"/etc/passwd" # Use standard library functions
create => "true",
comment => "Ensure listed users are present",
perms => mog("644","root","root"),
edit_line => append_users_starting("addpasswd.pwd");
}
Promises available in CFEngine
meta - information about promise bundles
Meta-data promises have no internal function. They are intended to be used to represent arbitrary information about promise bundles. Formally, meta promises are implemented as variables, and the values map to a variable context called bundlename_meta. The values can be used as variables and will appear in CFEngine Enterprise variable reports.
See meta
.
vars - a variable, representing a value
Variables in CFEngine are defined as promises that an identifier of a certain type represents a particular value. Variables can be scalars or lists of types string, int, real or data.
The allowed characters in variable names are alphanumeric (both upper and lower case) and underscore. Associative arrays using the string type and square brackets [] to enclose an arbitrary key are being deprecated in favor of the data variable type.
See vars
.
defaults - a default value for bundle parameters
Defaults promises are related to variables. If a variable or parameter in a promise bundle is undefined, or its value is defined to be invalid, a default value can be promised instead.
CFEngine does not use Perl semantics: i.e. undefined variables do not map to the empty string, they remain as variables for possible future expansion. Some variables might be defined but still contain unresolved variables. To handle this you will need to match the $(abc)
form of the variables.
See defaults
.
classes - a class, representing a state of the system
Classes promises may be made in any bundle. Classes that are set in common bundles are global in scope, while classes in all other bundles are local.
Note: The term class and context are sometimes used interchangeably.
See classes
.
users - add or remove users
User promises are promises made about local users on a host. They express which users should be present on a system, and which attributes and group memberships the users should have.
Every user promise has at least one attribute, policy, which describes whether or not the user should be present on the system. Other attributes are optional; they allow you to specify UID, home directory, login shell, group membership, description, and password.
A bundle can be associated with a user promise, such as when a user is created in order to do housekeeping tasks in his/her home directory, like putting default configuration files in place, installing encryption keys, and storing a login picture.
History: Introduced in CFEngine 3.6.0
See users
.
files - configure a file
Files promises are an umbrella for attributes of files. Operations fall basically into three categories: create, delete and edit.
See files
.
packages - install a package
CFEngine supports a generic approach to integration with native operating support for packaging. Package promises allow CFEngine to make promises regarding the state of software packages conditionally, given the assumption that a native package manager will perform the actual manipulations. Since no agent can make unconditional promises about another, this is the best that can be achieved.
See packages
.
guest_environments
Guest environment promises describe enclosed computing environments that can host physical and virtual machines, Solaris zones, grids, clouds or other enclosures, including embedded systems. CFEngine will support the convergent maintenance of such inner environments in a fixed location, with interfaces to an external environment.
CFEngine currently seeks to add convergence properties to existing interfaces for automatic self-healing of guest environments. The current implementation integrates with libvirt, supporting host virtualization for Xen, KVM, VMWare, etc. Thus CFEngine, running on a virtual host, can maintain the state and deployment of virtual guest machines defined within the libvirt framework. Guest environment promises are not meant to manage what goes on within the virtual guests. For that purpose you should run CFEngine directly on the virtual machine, as if it were any other machine.
See guest_environments
.
methods - take on a whole bundle of other promises
Methods are compound promises that refer to whole bundles of promises. Methods may be parameterized. Methods promises are written in a form that is ready for future development. The promiser object is an abstract identifier that refers to a collection (or pattern) of lower level objects that are affected by the promise-bundle. Since the use of these identifiers is for the future, you can simply use any string here for the time being.
See methods
.
processes - start or terminate processes
Process promises refer to items in the system process table, i.e., a command in some state of execution (with a Process Control Block). Promiser objects are patterns that are unanchored, meaning that they match line fragments in the system process table.
See processes
.
services - start or stop services
A service is a set of zero or more processes. It can be zero if the service is not currently running. Services run in the background, and do not require user intervention.
Service promises may be viewed as an abstraction of process and commands promises. An important distinguisher is however that a single service may consist of multiple processes. Additionally, services are registered in the operating system in some way, and get a unique name. Unlike processes and commands promises, this makes it possible to use the same name both when it is running and not.
Some operating systems are bundled with a lot of unused services that are running as default. At the same time, faulty or inherently insecure services are often the cause of security issues. With CFEngine, one can create promises stating the services that should be stopped and disabled.
The operating system may start a service at boot time, or it can be started by CFEngine. Either way, CFEngine will ensure that the service maintains the correct state (started, stopped, or disabled). On some operating systems, CFEngine also allows services to be started on demand, when they are needed. This is implemented though the inetd or xinetd daemon on Unix. Windows does not support this.
CFEngine also allows for the concept of dependencies between services, and can automatically start or stop these, if desired. Parameters can be passed to services that are started by CFEngine.
See services
.
commands - execute a command
Commands and processes are separated cleanly. Restarting of processes must be coded as a separate command. This stricter type separation allows for more careful conflict analysis to be carried out.
See commands
.
storage - verify attached storage
Storage promises refer to disks and filesystem properties.
See storage
.
databases - configure a database
CFEngine can interact with commonly used database servers to keep promises about the structure and content of data within them.
There are two main cases of database management to address: small embedded databases and large centralized databases.
Databases are often centralized entities that have a single point of management. While large monolithic database can be more easily managed with other tools, CFEngine can still monitor changes and discrepancies. In addition, CFEngine can also manage smaller embedded databases that are distributed in nature, whether they are SQL, registry or future types.
For example, creating 100 new databases for test purposes is a task for CFEngine; but adding a new item to an important production database is not a recommended task for CFEngine.
See databases
.
access - grant or deny access to file objects
Access promises are conditional promises made by resources living on the server.
The promiser is the name of the resource affected and is interpreted to be a path, unless a different resource_type is specified. Access is then granted to hosts listed in admit_ips, admit_keys and admit_hostnames, or denied using the counterparts deny_ips, deny_keys and deny_hostnames.
You layer the access policy by denying all access and then allowing it only to selected clients, then denying to an even more restricted set.
See access
.
roles - allow certain users to activate certain classes
Roles promises are server-side decisions about which users are allowed to define soft-classes on the server's system during remote invocation of cf-agent. This implements a form of Role Based Access Control (RBAC) for pre-assigned class-promise bindings. The user names cited must be attached to trusted public keys in order to be accepted. The regular expression is anchored, meaning it must match the entire name.
See roles
.
measurements - measure or sample data from the system
This is an Enterprise-only feature.
By default,CFEngine's monitoring component cf-monitord records performance data about the system. These include process counts, service traffic, load average and CPU utilization and temperature when available.
CFEngine Enterprise extends this in two ways. First it adds a three year trend summary based any 'shift'-averages. Second, it adds customizable measurements promises to monitor or log very specific user data through a generic interface. The end-result is to either generate a periodic time series, like the above mentioned values, or to log the results to custom-defined reports.
Promises of type measurement are written just like all other promises within a bundle destined for the agent concerned, in this case monitor. However, it is not necessary to add them to the bundlesequence, because cf-monitord executes all bundles of type monitor.
See measurements
.
reports - report a message
Reports promises simply print messages. Outputting a message without qualification can be a dangerous operation. In a large installation it could unleash an avalanche of messaging.
See reports
.
Authoring policy tools & workflow
There are several ways to approach authoring promises and ensuring they are copied into and then deployed properly from the masterfiles
directory:
- Create or modify files directly in the
masterfiles
directory. - Copy new or modified files into the
masterfiles
directory (e.g. local file copy usingcp
,scp
overssh
). - Utilize a version control system (e.g. Git) to push/pull changes or add new files to the
masterfiles
directory. - Utilize CFEngine Enterprise's integrated Git repository.
Authoring on a Workstation and Pushing to the Hub Using Git + GitHub
General Summary
- The "masterfiles" directory contains the promises and other related files (this is true in all situations).
- Replace the out of the box setup with an initialized
git
repository and remote to a clone hosted on GitHub. - Add a promise to
masterfiles
that tells CFEngine to check thatgit
repository for changes, and if there are any to merge them intomasterfiles
. - When an author wants to create a new promise, or modify an existing one, they clone the same repository on GitHub so that they have a local copy on their own computer.
- The author will make their edits or additions in their local version of the
masterfiles
repository. - After the author is done making their changes commit them using
git commit
. - After the changes are committed they are then pushed back to the remote repository on GitHub.
- As described in step 3, CFEngine will pull any new changes that were pushed to GitHub (sometime within a five minute time interval).
- Those changes will first exist in
masterfiles
, and then afterwards will be deployed to CFEngine hosts that are bootstrapped to the hub.
Create a Repository on GitHub for Masterfiles
There are two methods possible with GitHub: one is to use the web interface at GitHub.com; the second is to use the GitHub application.
Method One: Create Masterfiles Repository Using GitHub Web Interface
1a. In the GitHub web interface, click on the New repository
button.
1b. Or from the +
drop down menu on the top right hand side of the screen select New repository
.
2. Fill in a value in the Repository name
text entry (e.g. cfengine-masterfiles).
3. Select private
for the type of privacy desired (public
is also possible, but is not recommended in most situations).
4. Optionally, check the Initialize this repository with a README
box. (not required):""
Method Two: Create Masterfiles Repository Using the GitHub Application
- Open the GitHub app and click on the "+ Create" sign to create a new repository.
- Fill in a value in the
Repository name
text entry (e.g. cfengine-masterfiles). - Select
private
for the type of privacy desired (public
is also possible, but is not recommended in most situations). - Select one of your "Accounts" where you want the new repository to be created.
- Click on the "Create" button at the bottom of the screen. A new repository will be created in your local GitHub folder.
Initialize Git Repository in Masterfiles on the Hub
cd /var/cfengine/masterfiles
echo cf_promises_validated >> .gitignore
echo cf_promises_release_id >> .gitignore
git init
git commit -m "First commit"
git remote add origin https://github.com/GitUserName/cfengine-masterfiles.git
git push -u origin master
Note: cf_promises_validated
and cf_promises_release_id
should be explicitly excluded from VCS as shown above. They are generated files and involved in controlling policy updates. If these files are checked into the repository it can create issues with policy distribution.
Using the above steps on a private repository will fail with a 403 error. There are different approaches to deal with this:
A) Generate a key pair and add it to GitHub
- As root, type
ssh-keygen -t rsa
. - Hit enter when prompted to
Enter file in which to save the key (/root/.ssh/id_rsa):
. - Hit enter again when prompted to
Enter passphrase (empty for no passphrase):
. - Type
ssh-agent bash
and then the enter key. - Type
ssh-add /root/.ssh/id_rsa
. - Type
exit
to leavessh-agent bash
. - To test, type
ssh -T git@github.com
. - Open the generated key file (e.g.
vi /root/.ssh/id_rsa.pub
). - Copy the contents of the file to the clipboard (e.g. Ctrl+Shift+C).
- In the GitHub web interface, click the user account settings button (the icon with the two tools in the top right hand corner).
- On the next screen, on the left hand side, click
SSH keys
. - Click
Add SSH key
on the next screen. - Provide a
Title
for the label (e.g. CFEngine). - Paste the key contents from the clipboard into the
Key
textarea. - Click
Add key
. - If prompted to do so, provide your GitHub password, and then click the
Confirm
button.
B) Or, change the remote url to https://GitUserName@password:github.com/GitUserName/cfengine-masterfiles.git
. This is not safe in a production environment and should only be used for basic testing purposes (if at all).
Create a Remote in Masterfiles on the Hub to Masterfiles on GitHub
- Change back to the
masterfiles
directory, if not already there:
cd /var/cfengine/masterfiles
- Create the remote using the following pattern:
git remote add upstream ssh://git@github.com/GitUserName/cfengine-masterfiles.git
- Verify the remote was registered properly:
git remote -v
* You will see the remote definition in a list alongside any other previously defined remote entries.
Add a Promise that Pulls Changes to Masterfiles on the Hub from Masterfiles on GitHub
- Create a new file in
/var/cfengine/masterfiles
with a unique filename (e.g.vcs_update.cf
) - Add the following text to the
vcs_update.cf
file:
bundle agent vcs_update
{
commands:
"/usr/bin/git"
args => "pull --ff-only upstream master",
contain => masterfiles_contain;
}
body contain masterfiles_contain
{
chdir => "/var/cfengine/masterfiles";
}
- Save the file.
- Add bundle and file information to
/var/cfengine/masterfiles/promises.cf
. Example (where...
represents existing text in the file, omitted for clarity):
body common control
{
bundlesequence => {
...
vcs_update,
};
inputs => {
...
"vcs_update.cf",
};
- Save the file.
Editors
Using an editor that provides syntax highlighting and other features can significantly enhance prodcutivity and quality of life.
Emacs
For Emacs users, editing CFEngine policies is easy with the built-in CFEngine 3 mode in the cfengine.el library. For an overview of the capabilities, see the webinar by Ted Zlatanov and Appendix A of Diego Zamboni's Learning CFEngine book.
Spacemacs
Spacemacs has a CFEngine layer that configures many of the features shown in the Emacs webinar above out of the box as well as integration for executing cfengine3
src
blocks in org-mode
. It's a great way for vi/vim
lovers to leverage the power of Emacs.
Vi/Vim
Vi/Vim users can edit CFEngine policies with Neil Watson's CFEngine 3 scripts, available as GPL-software on GitHub. Neil's vi mode is also described in Appendix B of Diego Zamboni's "Learning CFEngine" book.
Visual Studio Code
Microsoft VS Code users have syntax highlighting thanks to AZaugg. Install the syntax highlighting and snippets directly from within Visual Studio Code by running ext install vscode-cfengine.
Sublime Text
Sublime Text 2 and 3 users have syntax highlighting and snippets thanks to Valery Astraverkhau. Get the syntax highlighting and snippets from his github repository. Aki Vanhatalo has contributed a beautifier to automatically re-indent policy in Sublime Text. Sublime Screenshot
Atom
Using Githubs hackable editor? You can get syntax highlighting with the language-cfengine3 package.
Eclipse
Interested in syntax highlighting for your CFEngine policy in Eclipse? Try this contributed syntax definition.
Want more out of your Eclipse & CFEngine experience? Itemis Xtext expert Boris Holzer developed a CFEngine workbench for Eclipse. They even published a brief screen-cast highlighting many of its features. For more information about their workbench please contact them using this form.
Kate
Users of the editor of the KDE desktop, Kate, have syntax highlighting available thanks to Jessica Greer, John Coleman, and Panos Christeas. The syntax highlighting definition can be found with the CFEngine source in contrib.
Policy style guide
Style is a very personal choice and the contents of this guide should only be considered suggestions. We invite you to contribute to the growth of this guide.
Style summary
- One indent = 2 spaces
- Avoid letting line length surpass 80 characters.
- When writing policy for documentation / blog posts / tutorials: Try to split up lines and fit within 45 characters in general, as long as it's not too problematic. (This will avoid horizontal scrolling on small windows and mobile).
- Add one indentation level per nesting of logic you are inside (promise type, class guard, promise, parenthesis, curly brace);
- Macros (
@if
etc.): 0 indents (never indented) - Promise types: 1 indent
- Class guards: +1 indent (2 indents in bundle, 1 indent in body)
- Promisers: +1 indent (2 or 3 indents, depending on whether there is a class guard or not)
- Promise attributes: +1 indent from promiser (3 or 4 indents).
- Parentheses: +1 indent (function calls or bundle invokations across multiple lines).
- Curly braces: +1 indent (slists / JSON / data containers).
- Macros (
Promise ordering
There are two common styles that are used when writing policy. The Normal Order style dictates that promises should be written in in the Normal Order that the agent evaluates promises in. The other is reader optimized where promises are written in the order they make sense to the reader. Both styles have their merits, but there seems to be a trend toward the reader optimized style.
1) Normal Order
Here is an example of a policy written in the Normal Order. Note how
packages
are listed after files
. This could confuse a novice who
thinks that it is necessary for the files promise to only be attempted
after the package promise is kept. However this style can be useful to
a policy expert who is familiar with Normal ordering.
bundle agent main
{
vars:
"sshd_config"
string => "/etc/ssh/sshd_config";
files:
"$(sshd_config)"
edit_line => insert_lines("PermitRootLogin no"),
classes => results("bundle", "sshd_config");
packages:
"ssh"
policy => "present";
package_module => apt_get;
services:
sshd_config_repaired::
"ssh"
service_policy => "restart",
comment => "After the sshd config file has been repaired, the
service must be reloaded in order for the new
settings to take effect.";
}
2) Reader Optimized
Here is an example of a policy written to be optimized for the reader. Note how packages are listed before files in the order which users think about taking imperitive action. This style can make it significantly easier for a novice to understand the desired state, but it is important to remember that Normal ordering still applies and that the promises will not be actuated in the order they are written.
bundle agent main
{
vars:
"sshd_config"
string => "/etc/ssh/sshd_config";
packages:
"ssh"
policy => "present";
package_module => apt_get;
files:
"$(sshd_config)"
edit_line => insert_lines("PermitRootLogin no"),
classes => results("bundle", "sshd_config");
services:
sshd_config_repaired::
"ssh"
service_policy => "restart",
comment => "After the sshd config file has been repaired, the
service must be reloaded in order for the new
settings to take effect.";
}
Whitespace and line length
Spaces are preferred to tab characters. Lines should not have trailing whitespace. Generally line length should not surpass 80 characters.
Curly brace alignment
The curly braces for top level blocks (bundle, body, promise) should be on separate lines. Content inside should be indented one level.
Example:
bundle agent example
{
vars:
"people"
slist => {
"Obi-Wan Kenobi",
"Luke Skywalker",
"Chewbacca",
"Yoda",
"Darth Vader",
};
"cuddly" slist => { "Chewbacca", "Yoda" };
}
Promise types
Promise types should have 1 indent and each promise type after the first listed should have a blank line before the next promise type.
This example illustrates the blank line before the "classes" type.
bundle agent example
{
vars:
"policyhost" string => "MyPolicyServerHostname";
classes:
"EL5" or => { "centos_5", "redhat_5" };
"EL6" or => { "centos_6", "redhat_6" };
}
Class guards
Class guards (sometimes called context class expressions) should have +1 indent, meaning 2 indents in bundles, and 1 indent in bodies.
The implicit any::
class guard can be added to more clearly mark what belongs to which class guard:
Example:
bundle agent example
{
vars:
any::
"foo" string => "bar";
windows::
"foo" string => "baz";
any::
"fizz" string => "buzz";
}
Single line and multi line promises
Promises with 1 (or 0) attributes may be put on a single line as long as they fit within 80 characters. Often it can improve readability to split up a promise into multiple lines, even if it does not exceed 80 characters.
Promises with multiple attributes should never be put on a single line. Promises over multiple lines should always have the attributes on separate lines. Examples:
bundle agent example
{
vars:
# Short promises can be on one line:
"a" string => "foo";
# Small lists are also okay:
"b" slist => { "1", "2", "3" };
# Don't put multiple attributes on one line:
"c" string => "foo", comment => "bar";
# Not like this either:
"c"
string => "foo", comment => "bar";
# Split up instead:
"c"
string => "foo",
comment => "bar";
# When splitting up, don't keep the attribute name on the same line:
"e" slist => {
"lorem ipsum dolor sit",
"foo bar baz",
"fizz buzz fizzbuzz",
};
# Instead, put the attribute name on a separate line:
"e"
slist => {
"lorem ipsum dolor sit",
"foo bar baz",
"fizz buzz fizzbuzz",
};
}
Policy comments
In-line policy comments are useful for debugging and explaining why something is done a specific way. We encourage you to document your policy thoroughly.
Comments about general body and bundle behavior and parameters should be placed after the body or bundle definition, before the opening curly brace and should not be indented. Comments about specific promise behavior should be placed before the promise at the same indention level as the promiser or on the same line after the attribute.
bundle agent example(param1)
# This is an example bundle to illustrate comments
# param1 - string -
{
vars:
"copy_of_param1" string => "$(param1)";
"jedi" slist => {
"Obi-Wan Kenobi",
"Luke Skywalker",
"Yoda",
"Darth Vader", # He used to be a Jedi, and since he
# tossed the emperor into the Death
# Star's reactor shaft we are including
# him.
};
classes:
# Most of the time we don't need differentiation of redhat and centos
"EL5" or => { "centos_5", "redhat_5" };
"EL6" or => { "centos_6", "redhat_6" };
}
Policy reports
It is common and useful to include reports in policy to get detailed information about what is going on. During a normal agent run the goal is to have 0 output so reports should always be guarded with a class. Carefully consider when your policy should generate report output. For policy degbugging type information (value of variables, classes that were set or not) the following style is recommended:
bundle agent example
{
reports:
DEBUG|DEBUG_example::
"DEBUG $(this.bundle): Desired Report Output";
}
As of version 3.7 variables can be used in double colon class expressions. If your policy will only be parsed by 3.7 or newer agents the following style is recommended:
bundle agent example
{
reports:
"DEBUG|DEBUG_$(this.bundle)"::
"DEBUG $(this.bundle): Desired Report Output";
}
Following this style keeps policy debug reports from spamming logs. It avoids
polluting the inform_mode
and verbose_mode
output, and it allows you to get
debug output for ALL policy or just a select bundle which is incredibly useful
when debugging a large policy set.
Promise handles
Promise handles uniquely identify a promise within a policy. We suggest a simple naming
scheme of bundle_name_promise_type_class_restriction_promiser
to keep handles unique and
easily identifiable. Often it may be easier to omit the handle.
bundle agent example
{
commands:
dev::
"/usr/bin/git"
args => "pull",
contain => in_dir("/var/srv/myrepo"),
if => "redhat",
handle => "example_commands_dev_redhat_git_pull";
}
Hashrockets (=>)
You may align hash rockets within a promise body scope and for grouped single line promises.
Example:
bundle agent example
{
files:
any::
"/var/cfengine/inputs/"
copy_from => update_policy( "/var/cfengine/masterfiles","$(policyhost)" ),
classes => policy_updated( "policy_updated" ),
depth_search => recurse("inf");
"/var/cfengine/modules"
copy_from => update_policy( "/var/cfengine/modules", "$(policyhost" ),
classes => policy_updated( "modules_updated" );
classes:
"EL5" or => { "centos_5", "redhat_5" };
"EL6" or => { "centos_6", "redhat_6" };
}
You may also simply leave them as they are:
bundle agent example
{
files:
any::
"/var/cfengine/inputs/"
copy_from => update_policy( "/var/cfengine/masterfiles","$(policyhost)" ),
classes => policy_updated( "policy_updated" ),
depth_search => recurse("inf");
"/var/cfengine/modules"
copy_from => update_policy( "/var/cfengine/modules", "$(policyhost" ),
classes => policy_updated( "modules_updated" );
classes:
"EL5" or => { "centos_5", "redhat_5" };
"EL6" or => { "centos_6", "redhat_6" };
}
Which one do you prefer?
Naming conventions
Naming conventions can also help to provide clarity.
Snakecase
Words delimited by an underscore. This style is prevalant for variables, classes, bundle and body names in the Masterfiles Policy Framework.
bundle agent __main__
{
methods:
"ssh";
}
bundle agent ssh
{
vars:
"service_name" string => "ssh";
"config_file" string => "/etc/ssh/sshd_config";
"conf[Port]" string => "22";
files:
"$(config_file)"
edit_line => default:set_line_based("$(this.bundle).conf",
" ",
"\s+",
".*",
"\s*#\s*"),
classes => default:results( "bundle", "$(config_file)");
services:
_etc_ssh_sshd_config_repaired::
"$(service_name)"
service_policy => "restart",
classes => default:results( "bundle", "$(service_name)_restart");
reports:
ssh_restart_repaired._etc_ssh_sshd_config_repaired::
"We restarted ssh because the config file was repaired";
}
This policy can be found in
/var/cfengine/share/doc/examples/style_snake_case.cf
and downloaded directly from
github.
Pascalecase
Words delimited by capital Letters.
bundle agent __main__
{
methods:
"Ssh";
}
bundle agent Ssh
{
vars:
"ServiceName" string => "ssh";
"ConfigFile" string => "/etc/ssh/sshd_config";
"Conf[Port]" string => "22";
files:
"$(ConfigFile)"
edit_line => default:set_line_based("$(this.bundle).Conf",
" ",
"\s+",
".*",
"\s*#\s*"),
classes => default:results( "bundle", "$(ConfigFile)");
services:
_etc_ssh_sshd_config_repaired::
"$(ServiceName)"
service_policy => "restart",
classes => default:results( "bundle", "$(ServiceName)_restart");
reports:
ssh_restart_repaired._etc_ssh_sshd_config_repaired::
"We restarted ssh because the config file was repaired";
}
This policy can be found in
/var/cfengine/share/doc/examples/style_PascaleCase.cf
and downloaded directly from
github.
Camelcase
Words are delimited by capital letters, except the initial word.
bundle agent __main__
{
methods:
"Ssh";
}
bundle agent ssh
{
vars:
"serviceName" string => "ssh";
"configFile" string => "/etc/ssh/sshd_config";
"conf[Port]" string => "22";
files:
"$(configFile)"
edit_line => default:set_line_based("$(this.bundle).conf",
" ",
"\s+",
".*",
"\s*#\s*"),
classes => default:results( "bundle", "$(configFile)");
services:
_etc_ssh_sshd_config_repaired::
"$(serviceName)"
service_policy => "restart",
classes => default:results( "bundle", "$(serviceName)_restart");
reports:
ssh_restart_repaired._etc_ssh_sshd_config_repaired::
"We restarted ssh because the config file was repaired";
}
This policy can be found in
/var/cfengine/share/doc/examples/style_camelCase.cf
and downloaded directly from
github.
Hungarian notation
Hungarian notation can
help improve the readability of policy, especially when working with lists and
data containers where the use of @
or $
significantly affects the behavior
of the policy.
bundle agent __main__
{
vars:
"s_one" string => "one";
"ITwo" int => "2";
"rThree" real => "3.0";
"lMyList" slist => { "$(s_one)", "$(ITwo)", "$(rThree)" };
methods:
"Iteration inside (bundle called once)"
usebundle => dollar_vs_at( @(lMyList) );
"Iteration outside (bundle called length(lMyList) times)"
usebundle => dollar_vs_at( $(lMyList) );
}
bundle agent dollar_vs_at( myParam )
{
vars:
"myParamType" string => type( myParam );
classes:
"myParamType_slist" expression => strcmp( $(myParamType), "slist" );
"myParamType_string" expression => strcmp( $(myParamType), "string" );
reports:
"Bundle promised by '$(with)'"
with => nth( reverse( callstack_promisers() ), 0 );
myParamType_slist::
"myParam is of type '$(myParamType)' with value $(with)"
with => join( ", ", @(myParam) );
myParamType_string::
"myParam is of type '$(myParamType)' with value $(myParam)";
}
R: Bundle promised by 'Iteration inside (bundle called once)'
R: myParam is of type 'slist' with value one, 2, 3.000000
R: Bundle promised by 'Iteration outside (bundle called length(lMyList) times)'
R: myParam is of type 'string' with value one
R: myParam is of type 'string' with value 2
R: myParam is of type 'string' with value 3.000000
This policy can be found in
/var/cfengine/share/doc/examples/style_hungarian.cf
and downloaded directly from
github.
Classes
Classes are intended to describe an aspect of the system, and they are combined in expressions to restrict when and where a promise should be actuated (class guards). To make this desired state easier to read classes should be named to describe the current state, not an action that should take place.
For example, here is a policy that uses a class that indicates an
action that should be taken after having repaired the sshd
config.
bundle agent main
{
vars:
"sshd_config" string => "/etc/ssh/sshd_config";
files:
"$(sshd_config)"
edit_line => insert_lines("PermitRootLogin no"),
classes => if_repaired("restart_sshd");
services:
!windows::
"ssh"
service_policy => "start",
comment => "We always want ssh to be running so that we have
administrative access";
restart_sshd::
"ssh"
service_policy => "restart",
comment => "Here it's kind of hard to tell *why* we are
restarting sshd";
}
Here is a slightly improved version that shows using classes to describe the
current state, or what happened as the result of the promise. Note how it's
easier to determine why the ssh service should be restarted. Using
the results
, scoped_classes_generic
, or
classes_generic
classes bodies can help improve class name consistency and are
highly recommended.
bundle agent main
{
vars:
"sshd_config"
string => "/etc/ssh/sshd_config";
files:
"$(sshd_config)"
edit_line => insert_lines("PermitRootLogin no"),
classes => results("bundle", "sshd_config");
services:
!windows::
"ssh"
service_policy => "start",
comment => "We always want ssh to be running so that we have
administrative access";
sshd_config_repaired::
"ssh"
service_policy => "restart",
comment => "After the sshd config file has been repaired, the
service must be reloaded in order for the new
settings to take effect.";
}
Deprecating bundles
As your policy library changes over time you may want to deprecate various bundles in favor of newer implimentations. To indicate that a bundle is deprecated we recommend the following style.
bundle agent old
{
meta:
"tags"
slist => {
"deprecated=3.6.0",
"deprecation-reason=More feature rich implimentation",
"replaced-by=newbundle",
};
}
Tooling
Currently, there is no canonical policy linting or reformatting tool. There are a few different tools that can be useful apart from an editor with syntax support for achieving regular formatting.
cf-promises
cf-promises
can output the parsed policy using the --policy-output-format
option. Beware, this will strip macros as they are done during parse time.
Example policy:
bundle agent satellite_bootstrap_main
{
@if feature(this_is_not_the_feature_your_looking_for)
Hello there.
@endif
meta:
(!ubuntu&!vvlan&!role_satellite)::
"tags" slist => { "autorun" };
methods:
"bootstrap rhel7 servers to satellite every 24 hours"
usebundle => satellite_bootstrap,
action => if_elapsed(1440);
}
Output the parsed policy in cf
format:
cf-promises -f /tmp/example.cf --policy-output-format cf
Formatted parsed policy:
bundle agent satellite_bootstrap_main()
{
meta:
(!ubuntu&!vvlan&!sarcrole_satellite)::
"tags" slist => {"autorun"};
methods:
any::
"bootstrap rhel7 servers to satellite every 24 hours"
usebundle => satellite_bootstrap,
action => if_elapsed("1440");
}
body file control()
{
inputs => { "$(sys.libdir)/stdlib.cf" };
}
CFEngine beautifier
Written as a package for the Sublime Text editor, the CFEngine Beautifier can also be used from the command line as a stand-alone tool.
reindent.pl
reindent.pl
is available from the contrib directory in the core repository. You can run
reindent.pl FILE1.cf FILE2.c FILE3.h
to reindent files, if you don't want to
set up Emacs. It will rewrite them with the new indentation, using Emacs in
batch mode.
Bundles best practices
The following contains practices to remember when creating bundles as you write policy.
How to choose and name bundles
Use the name of a bundle to represent a meaningful aspect of system administration, We recommend using a two- or three-part name that explains the context, general subject heading, and special instance. Names should be service-oriented in order to guide non-experts to understand what they are about.
For example:
- app_mail_postfix
- app_mail_mailman
- app_web_apache
- app_web_squid
- app_web_php
- app_db_mysql
- garbage_collection
- security_check_files
- security_check_processes
- system_name_resolution
- system_xinetd
- system_root_password
- system_processes
- system_files
- win_active_directory
- win_registry
- win_services
When to make a bundle
Put items into a single bundle if:
- They belong to the same conceptual aspect of system administration.
- They do not need to be switched on or off independently.
Put items into different bundles if:
- All of the promises in one bundle need to the checked before all of the promises in another bundle.
- You need to re-use the promises with different parameters.
In general, keep the number of bundles to a minimum. This is a knowledge-management issue. Clarity comes from differentiation, but only if the number of items is small.
When to use a paramaterized bundle or method
If you need to arrange for a managed convergent collection or sequence of promises that will occur for a list of (multiple) names or promisers, then use a bundle to simplify the code.
Write the promises (which may or may not be ordered) using a parameter for the different names, and then call the method passing the list of names as a parameter to reduce the amount of code.
bundle agent testbundle
{
vars:
"userlist" slist => { "mark", "jeang", "jonhenrik", "thomas", "eben" };
methods:
"any" usebundle => subtest("$(userlist)");
}
###########################################
bundle agent subtest(user)
{
commands:
"/bin/echo Fix $(user)";
files:
"/home/$(user)/."
create => "true";
reports:
linux::
"Finished doing stuff for $(user)";
}
When to use classes in common bundles
- When you need to use them in multiple bundles (because classes defined in common bundles have global scope).
When to use variables in common bundles
- For rationality, if the variable does not belong to any particular bundle, because it is
used elsewhere. (Qualified variable names such as
$(mybundle.myname)
are always globally accessible, so this is a cosmetic issue.)
When to use variables in local bundles
- If they are not needed outside the bundles.
- If they are used for iteration (without qualified scope).
- If they are tied to a specific aspect of system maintenance represented by the bundle, so
that accessing
$(bundle.var)
adds clarity.
External data
It is common to integrate CFEngine with external data sources. External data sources could be hand edited data files, the cached result of an API call or generated by other tooling. This is especially useful for integrating CFEngine with other infrastructure components like a CMDB.
CFEngine can load structured data defined in JSON, YAML, CSV using the
readjson()
, readyaml()
, readcsv()
and readdata()
functions or by custom
parsing with data_resdstringarray()
and data_readstringarrayidx()
.
Additionally CFEngine provides the augments file as a way to define variables and classes that are available from the beginning of policy evaluation.
The augments file can be distributed globally as part of your policy by creating
def.json
in the root of your masterfiles, or it could be generated by the
agent itself for use in subsequent runs.
If the augments file is generated on the agent itself we recommend doing so from
a separate policy like update.cf
.
Testing policies
One of the practical advantages of CFEngine is that you can test it without the need for root or administrator privileges. This is useful if you are concerned about manipulating important system files, but naturally limits the possibilities for what CFEngine is able to do.
CFEngine operates with the notion of a work-directory. The default work
directory for the root
user is /var/cfengine
. For any other user, the work
directory lies in the user's home directory, named ~/.cfagent
.
CFEngine prefers you to keep certain files here. You should not resist this too strongly or you will make unnecessary trouble for yourself. The decision to have this 'known directory' was made to simplify a lot of configuration.
To test CFEngine as an ordinary user, do the following:
Copy the binaries into the work directory:
mkdir -p ~/.cfagent/inputs
mkdir -p ~/.cfagent/bin
cp /var/cfengine/bin/cf-* ~/.cfagent/bin
cp /var/cfengine/inputs/*.cf ~/.cfagent/inputs
You can test the software and play with configuration files by editing the
basic directly in the ~/.cfagent/inputs
directory. For example, try the
following:
console
~/.cfagent/bin/cf-promises
~/.cfagent/bin/cf-promises --verbose
This is always the way to start checking a configuration in CFEngine 3. If a
configuration does not pass this check/test, you will not be allowed to use
it, and cf-agent
will look for the file failsafe.cf
.
Controlling frequency
By default CFEngine runs relatively frequently (every 5 minutes) but you may not want every promise to be evaluated each agent execution. Classes and promise locks are the two primary ways in which a promises frequency can be controlled. Classes are the canonical way of controlling if a promise is in context and should be evaluated. Promise locks control frequency based on the number of minutes since the last promise actuation.
Controlling frequency using classes
Classes are the canonical way of controlling promise executions in CFEngine.
Use time based classes to restrict promises to run during a specific period of time. For example, here sshd
promises to be the latest version available, but only on Tuesdays during the first 15 minutes of the 5:00 hour.
bundle agent __main__
{
packages:
Tuesday.Hr05_Q1::
"sshd"
version => "latest",
comment => "Make sure sshd is at the latest version, but only Tuesday between 5:00 and 5:15am";
}
Persistent classes can exist for a period of time, across multiple executions of
cf-agent
. Persistent classes can be used to avoid re-execution of a promise.
For example, here /tmp/heartbeat.dat
promises to update it's timestamp when
heartbeat_repaired
is not defined. When the file is repaired the class
heartbeat_repaired
is defined for 10 minutes causing the promise to be out of
context during subsequent executions for the next 10 minutes.
bundle agent __main__
{
files:
!heartbeat_repaired::
"/tmp/heartbeat.dat"
create => "true",
touch => "true",
classes => persistent_results( "heartbeat", 10 );
}
body classes persistent_results( prefix, time )
{
inherit_from => results( "namespace", "$(prefix)" );
persist_time => "$(time)";
}
Controlling frequency using promise locks
CFEngine incorporates a series of locks which prevent it from checking promises too often, and which prevent it from spending too long trying to check promises it has recently verified. This locking mechanism works in such a way that you can start several CFEngine components simultaneously without them interfering with each other. You can control two things about each kind of action in CFEngine:
ifelapsed
- The minimum time (in minutes) which should have passed since the last time that promise was verified. It will not be executed again until this amount of time has elapsed. If the value is0
the promise has no lock and will always be executed when in context. Additionally, a value of0
disables function caching. Default time is1
minute.expireafter
- The maximum amount (in minutes) of timecf-agent
should wait for an old instantiation to finish before killing it and starting again. You can think aboutexpireafter
as a timeout to use when a promise verification may involve an operation that could wait indefinitely. Default time is120
minutes.
You can set these values either globally (for all actions) or for each action
separately. If you set global and local values, the local values override the
global ones. All times are written in units of minutes. The following global
setting is defined in body agent control
.
body agent control
{
ifelapsed => "60"; # one hour
}
This setting tells CFEngine not to verify promises until 60 minutes have
elapsed, ie ensures that the global frequency for all promise verification is
one hour. This global setting of one hour could be changed for a specific
promise body by setting ifelapsed
in the promise body.
body action example
{
ifelapsed => "90"; # 1.5 hours
}
This promise which overrides the global 60 minute time period and defines a frequency of 90 minutes.
These locks do not prevent the whole of cf-agent
from running, only
atomic promise checks on the same objects (packages, users, files,
etc.). Several different cf-agent
instances can run concurrently.
The locks ensure that promises will not be verified by two cf-agents
at the same time or too soon after a verification.
For example, here the sshd
package promises to be at the latest version. It
has the if_elapsed_day
action body attached which sets ifelapsed
to 1440
causing the promise lock to persist for a day effectively restricting the
promise to run just once a day.
bundle agent __main__
{
packages:
"sshd"
version => "latest",
action => if_elapsed_day,
comment => "Make sure sshd is at the latest version, but just once a day.";
}
Note:
- Promise locks are ignored when CFEngine is run with the
--no-lock
or-K
option, e.g. a common manual execution of the agent,cf-agent -KI
would not respect promises that are locked from a recent execution. - Locks are purged based on database utilization and age in order to maintain the integrity and health of the underlying lock database.
See also: cf_lock.lmdb