Centralized management

These examples show a simple setup for starting with a central approach to management of servers. Centralization of management is a simple approach suitable for small environments with few requirements. It is useful for clusters where systems are all alike.

All hosts the same

This shows the simplest approach in which all hosts are the same. It is too simple for most environments, but it serves as a starting point. Compare it to the next section that includes variation.

code
body common control
{
      bundlesequence  => { "central" };
}


bundle agent central
{
  vars:
      "policy_server" string => "myhost.domain.tld";
      "mypackages" slist => {
                              "nagios"
                              "gcc",
                              "apache2",
                              "php5"
      };

  files:
      # Password management can be very simple if all hosts are identical

      "/etc/passwd"
      comment   => "Distribute a password file",
      perms     => mog("644","root","root"),
      copy_from => secure_cp("/home/mark/LapTop/words/RoadAhead","$(policy_server)");

  packages:
      "$(mypackages)"
      package_policy => "add",
      package_method => generic;

      # Add more promises below ...

}


body server control
{
      allowconnects         => { "127.0.0.1" , "::1", "10.20.30.0/24" };
      allowallconnects      => { "127.0.0.1" , "::1", "10.20.30.0/24" };
      trustkeysfrom         => { "127.0.0.1" , "::1", "10.20.30.0/24" };
      # allowusers
}

bundle server my_access_rules()
{
  access:
      # myhost.domain.tld makes this file available to 10.20.30*

    myhost_domain_tld::
      "/etc/passwd"
      admit   => { "127.0.0.1", "10.20.30.0/24" };
}

Variation in hosts

code
body common control
{
      bundlesequence  => { "central" };
}


bundle agent central
{
  classes:
      "mygroup_1" or => { "myhost", "host1", "host2", "host3" };
      "mygroup_2" or => { "host4", "host5", "host6" };

  vars:
      "policy_server" string => "myhost.domain.tld";
    mygroup_1::
      "mypackages" slist => {
                              "nagios"
                              "gcc",
                              "apache2",
                              "php5"
      };
    mygroup_2::
      "mypackages" slist => {
                              "apache"
                              "mysql",
                              "php5"
      };

  files:
      # Password management can be very simple if all hosts are identical

      "/etc/passwd"
      comment   => "Distribute a password file",
      perms     => mog("644","root","root"),
      copy_from => secure_cp("/etc/passwd","$(policy_server)");

  packages:
      "$(mypackages)"
      package_policy => "add",
      package_method => generic;

      # Add more promises below ...

}


body server control
{
      allowconnects         => { "127.0.0.1" , "::1", "10.20.30.0/24" };
      allowallconnects      => { "127.0.0.1" , "::1", "10.20.30.0/24" };
      trustkeysfrom         => { "127.0.0.1" , "::1", "10.20.30.0/24" };
      # allowusers
}

bundle server my_access_rules()
{
  access:
      # myhost.domain.tld makes this file available to 10.20.30*

    myhost_domain_tld::
      "/etc/passwd"
      admit   => { "127.0.0.1", "10.20.30.0/24" };
}

Updating from a central hub

The configuration bundled with the CFEngine source code contains an example of centralized updating of policy that covers more subtleties than this example, and handles fault tolerance. Here is the main idea behind it. For simplicity, we assume that all hosts are on network 10.20.30.* and that the central policy server/hub is 10.20.30.123.

code
bundle agent update
{
  vars:
      "master_location" string => "/var/cfengine/masterfiles";

      "policy_server"   string => "10.20.30.123",
      comment => "IP address to locate your policy host.";

  files:
      "$(sys.workdir)/inputs"
      perms => system("600"),
      copy_from => remote_cp("$(master_location)",$(policy_server)),
      depth_search => recurse("inf");

      "$(sys.workdir)/bin"
      perms => system("700"),
      copy_from => remote_cp("/usr/local/sbin","localhost"),
      depth_search => recurse("inf");
}

body server control
{
      allowconnects         => { "127.0.0.1" , "10.20.30.0/24" };
      allowallconnects      => { "127.0.0.1" , "10.20.30.0/24" };
      trustkeysfrom         => { "127.0.0.1" , "10.20.30.0/24" };
}

bundle server my_access_rules()
{
  access:
    10_20_30_123::
      "/var/cfengine/masterfiles"
      admit   => { "127.0.0.1", "10.20.30.0/24" };
}

Laptop support configuration

Laptops do not need a lot of confguration support. IP addresses are set by DHCP and conditions are changeable. But you want to set your DNS search domains to familiar settings in spite of local DHCP configuration, and another useful trick is to keep a regular backup of disk changes on the local disk. This won't help against disk destruction, but it is a huge advantage when your user accidentally deletes files while travelling or offline.

code
body common control
{
    bundlesequence  => {
        "update",
        "garbage_collection",
        "main",
        "backup",
    };
    inputs => {
        "update.cf",
        "site.cf",
        "library.cf"
    };
}

body agent control
{
    # if default runtime is 5 mins we need this for long jobs
    ifelapsed => "15";
}

body monitor control
{
    forgetrate => "0.7";
}

body executor control
{
    splaytime => "1";
    mailto => "mark@iu.hio.no";
    smtpserver => "localhost";
    mailmaxlines => "30";
    # Instead of a separate update script, now do this

    exec_command => "$(sys.workdir)/bin/cf-agent -f failsafe.cf && $(sys.workdir)/bin/cf-agent";
}

bundle agent main
{
  vars:
    "component" slist => { "cf-monitord", "cf-serverd" };
      # - - - - - - - - - - - - - - - - - - - - - - - -

  files:
    "$(sys.resolv)"  # test on "/tmp/resolv.conf" #
      create        => "true",
      edit_line     => resolver,
      edit_defaults => def;

  processes:
    "$(component)" restart_class => canonify("$(component)_not_running");
      # - - - - - - - - - - - - - - - - - - - - - - - -

  commands:
    "$(sys.workdir)/bin/$(component)"
      if => canonify("$(component)_not_running");
}

bundle agent backup
{
  files:
    "/home/backup"
      copy_from => cp("/home/mark"),
      depth_search => recurse("inf"),
      file_select => exclude_files,
      action => longjob;
}

bundle agent garbage_collection
{
  files:
    "$(sys.workdir)/outputs"
      delete => tidy,
      file_select => days_old("3"),
      depth_search => recurse("inf");

}

Process management

code
body common control
{
    bundlesequence => { "test" };
}

bundle agent test
{
  processes:
    "sleep"
      signals => { "term", "kill" };
}

body common control
{
    bundlesequence => { "testbundle" };
}

bundle agent testbundle
{
  processes:
    "sleep"
      process_count   => up("sleep");
  reports:
    sleep_out_of_control::
      "Out of control";
}

body process_count up(s)
{
    match_range         => "5,10"; # or irange("1","10");
    out_of_range_define => { "$(s)_out_of_control" };
}

body common control
{
    bundlesequence => { "testbundle" };
}

bundle agent testbundle
{
  processes:
    ".*"
      process_select => proc_finder("a.*"),
      process_count  => up("cfservd");
}

body process_count up(s)
{
    match_range         => "1,10"; # or irange("1","10");
    out_of_range_define => { "$(s)_out_of_control" };
}

body process_select proc_finder(p)
{
    stime_range    => irange(ago("0","0","0","2","0","0"),now);
    process_result => "stime";
}


body common control
{
    bundlesequence => { "testbundle" };
}

bundle agent testbundle
{
  processes:
    ".*"
      process_select  => proc_finder("a.*"),
      process_count   => up("cfservd");
}

body process_count up(s)
{
    match_range         => "1,10"; # or irange("1","10");
    out_of_range_define => { "$(s)_out_of_control" };
}

body process_select proc_finder(p)
{
    process_owner  => { "avahi", "bin" };
    command        => "$(p)";
    pid            => "100,199";
    vsize          => "0,1000";
    process_result => "command.(process_owner|vsize)";
}

body common control
{
    bundlesequence => { "process_restart" };
}

bundle agent process_restart
{
  processes:
    "/usr/bin/daemon"
      restart_class => "launch";

  commands:
    launch::
      "/usr/bin/daemon";
}
body common control
{
    bundlesequence => { "process_restart" };
}

bundle agent process_restart
{
  vars:
    "component"
      slist => {
        "cf-monitord",
        "cf-serverd",
        "cf-execd"
      };

  processes:
    "$(component)"
      restart_class => canonify("$(component)_not_running");

  commands:
    "/var/cfengine/bin/$(component)"
      if => canonify("$(component)_not_running");
}

body common control
{
    bundlesequence => { "testbundle" };
}

bundle agent testbundle
{
  processes:

    "cfservd"
      process_count => up("cfservd");

    cfservd_out_of_control::
      "cfservd"
        signals       => { "stop" , "term" },
        restart_class => "stopped_out_of_control_cf_serverd";

  commands:
    stopped_out_of_control_cf_serverd::
      "/usr/local/sbin/cfservd";
}

body process_count up(s)
{
    match_range => "1,10"; # or irange("1","10");
    out_of_range_define => { "$(s)_out_of_control" };
}

Kill process

code
body common control
{
      bundlesequence => { "test" };
}

bundle agent test
{
  processes:

      "sleep"
      signals => { "term", "kill" };
}

Restart process

A basic pattern for restarting processes:

code
body common control
{
      bundlesequence => { "process_restart" };
}



bundle agent process_restart
{
  processes:

      "/usr/bin/daemon"
      restart_class => "launch";

  commands:

    launch::
      "/usr/bin/daemon";

}

This can be made more sophisticated to handle generic lists:

code
body common control
{
    bundlesequence => { "process_restart" };
}



bundle agent process_restart
{
  vars:

    "component"
      slist => {
        "cf-monitord",
        "cf-serverd",
        "cf-execd"
      };

  processes:

    "$(component)"
      restart_class => canonify("not_running_$(component)");

  commands:

    "/var/cfengine/bin/$(component)"
      if => canonify("not_running_$(component)");

}

Why? Separating this into two parts gives a high level of control and conistency to CFEngine. There are many options for command execution, like the ability to run commands in a sandbox or as setuid. These should not be reproduced in processes.

Mount a filesystem

code
body common control
{
      bundlesequence => { "mounts" };
}

bundle agent mounts
{
  storage:
      "/mnt" mount  => nfs("slogans.iu.hio.no","/home");
}

body mount nfs(server,source)
{
      mount_type => "nfs";
      mount_source => "$(source)";
      mount_server => "$(server)";
      #mount_options => { "rw" };
      edit_fstab => "true";
      unmount => "true";
}

Manage a system process

code
Ensure running
Ensure not running
Prune processes

Ensure running

The simplest example might look like this:

code
bundle agent restart_process
{
  processes:

      "httpd"

      comment => "Make sure apache web server is running",
      restart_class => "restart_httpd";

  commands:

    restart_httpd::

      "/etc/init.d/apache2 restart";

}

This example shows how the CFEngine components could be started using a pattern.

code
bundle agent CFEngine_processes
{
  vars:

    "component" slist => { "cf-execd", "cf-monitord", "cf-serverd", "cf-hub" };

  processes:

    "$(component)"
      comment => "Make sure server parts of CFEngine are running",
      restart_class => canonify("$(component)_not_running");

  commands:

    "$(sys.workdir)/bin/$(component)"
      comment => "Make sure server parts of CFEngine are running",
      if => canonify("$(component)_not_running");

}

Ensure not running

code
bundle agent restart_process
{
  vars:

      "killprocs" slist => { "snmpd", "gameserverd", "irc", "crack" };

  processes:

      "$(killprocs)"

      comment => "Ensure processes are not running",
      signals => { "term", "kill" };
}

Prune processes

This example kills processes owned by a particular user that have exceeded 100000 bytes of resident memory.

code
body common control
{
      bundlesequence  => { "testbundle"  };
}


bundle agent testbundle
{
  processes:

      ".*"

      process_select  => big_processes("mark"),
      signals => { "term" };
}


body process_select big_processes(o)
{
      process_owner => { $(o) };
      rsize => irange("100000","900000");
      process_result => "rsize.process_owner";
}

Set up HPC clusters

HPC cluster machines are usually all identical, so the CFEngine configuration is very simple. HPC clients value CPU and memory resources, so we can shut down unnecessary services to save CPU. We can also change the scheduling rate of CFEngine to run less frequently, and save a little:

code
body executor control
{
      splaytime => "1";
      mailto => "cfengine@example.com";
      smtpserver => "localhost";
      mailmaxlines => "30";
      # Once per hour, on the hour

      schedule     => { "Min00" };
}

bundle agent services_disable
{
  vars:
      # list all of xinetd services (case sensitive)

      "xinetd_services" slist => {
                                   "imap",
                                   "imaps",
                                   "ipop2",
                                   "ipop3",
                                   "krb5-telnet",
                                   "klogin",
                                   "kshell",
                                   "ktalk",
                                   "ntalk",
                                   "pop3s",
      };
  methods:
      # perform the actual disable all xinetd services according to the list above

      "any"  usebundle => disable_xinetd("$(xinetd_services)");

  processes:
      "$(xinetd_services)"
      signals => { "kill" };
}

bundle agent disable_xinetd(name)
{
  vars:
      "status" string => execresult("/sbin/chkconfig --list $(name)", "useshell");

  classes:
      "on"  expression => regcmp(".*on.*","$(status)");

  commands:
    on::
      "/sbin/chkconfig $(name) off",
      comment => "disable $(name) service";

  reports:
    on::
      "disable $(name) service.";
}

Set up name resolution

There are many ways to do name resolution setup1 We write a reusable bundle using the editing features.

A simple and straightforward approach is to maintain a separate modular bundle for this task. This avoids too many levels of abstraction and keeps all the information in one place. We implement this as a simple editing promise for the /etc/resolv.conf file.

code
bundle agent system_files
{
  files:
      "$(sys.resolv)"  # test on "/tmp/resolv.conf" #
      comment       => "Add lines to the resolver configuration",
      create        => "true",
      edit_line     => resolver,
      edit_defaults => std_edits;
      # ...other system files ...

}

bundle edit_line resolver
{
  delete_lines:
      # delete any old name servers or junk we no longer need

      "search.*";
      "nameserver 80.65.58.31";
      "nameserver 80.65.58.32";
      "nameserver 82.103.128.146";
      "nameserver 78.24.145.4";
      "nameserver 78.24.145.5";
      "nameserver 128.39.89.10";

  insert_lines:
      "search mydomain.tld" location => start;
    special_net::
      "nameserver 128.39.89.8";
      "nameserver 128.39.74.66";
    !special_net::
      "nameserver 128.38.34.12";
    any::
      "nameserver 212.112.166.18";
      "nameserver 212.112.166.22";
}

A second approach is to try to conceal the operational details behind a veil of abstraction.

code
bundle agent system_files
{
  vars:
      "searchlist"  string => "iu.hio.no CFEngine.com";
      "nameservers" slist => {
                               "128.39.89.10",
                               "128.39.74.16",
                               "192.168.1.103"
      };

  files:
      "$(sys.resolv)"  # test on "/tmp/resolv.conf" #
      create        => "true",
      edit_line     => doresolv("$(s)","@(this.n)"),
      edit_defaults => empty;
      # ....

}

bundle edit_line doresolv(search,names)
{
  insert_lines:
      "search $(search)";
      "nameserver $(names)";
}

bundle agent system_files { # ...

files: "/etc/hosts" comment => "Add hosts to the /etc/hosts file", edit_line => fix_etc_hosts; }

bundle edit_line fix_etc_hosts { vars: "names[127.0.0.1]" string => "localhost localhost.CFEngine.com"; "names[128.39.89.12]" string => "myhost myhost.CFEngine.com"; "names[128.39.89.13]" string => "otherhost otherhost.CFEngine.com"; # etc

code
  "i" slist => getindices("names");

insert_lines: "$(i) $(names[$(i)])"; } ```

DNS is not the only name service, of course. Unix has its older /etc/hosts file which can also be managed using file editing. We simply append this to the system_files bundle.

code
bundle agent system_files
{
  vars:
      "searchlist"  string => "iu.hio.no CFEngine.com";
      "nameservers" slist => {
                               "128.39.89.10",
                               "128.39.74.16",
                               "192.168.1.103"
      };

  files:
      "$(sys.resolv)"  # test on "/tmp/resolv.conf" #
      create        => "true",
      edit_line     => doresolv("$(s)","@(this.n)"),
      edit_defaults => empty;
      # ....

}

bundle edit_line doresolv(search,names)
{
  insert_lines:
      "search $(search)";
      "nameserver $(names)";
}

bundle agent system_files { # ...

files: "/etc/hosts" comment => "Add hosts to the /etc/hosts file", edit_line => fix_etc_hosts; }

bundle edit_line fix_etc_hosts { vars: "names[127.0.0.1]" string => "localhost localhost.CFEngine.com"; "names[128.39.89.12]" string => "myhost myhost.CFEngine.com"; "names[128.39.89.13]" string => "otherhost otherhost.CFEngine.com"; # etc

code
  "i" slist => getindices("names");

insert_lines: "$(i) $(names[$(i)])"; } ```

Set up sudo

Setting up sudo is straightforward, and is best managed by copying trusted files from a repository.

code
bundle agent system_files
{
  vars:
      "masterfiles" string => "/subversion_projects/masterfiles";
      # ...

  files:
      "/etc/sudoers"
      comment => "Make sure the sudo configuration is secure and up to date",
      perms => mog("440","root","root"),
      copy_from => secure_cp("$(masterfiles)/sudoers","$(policy_server)");
}

Environments (virtual)

code
body common control
{
      bundlesequence  => { "my_vm_cloud" };
}

bundle agent my_vm_cloud
{
  vars:
      "vms[atlas]" slist => { "guest1", "guest2", "guest3" };

  environments:
    scope||any::  # These should probably be in class "any" to ensure uniqueness
      "$(vms[$(sys.host)])"
      environment_resources => virt_xml("$(xmlfile[$(this.promiser)])"),
      environment_interface => vnet("eth0,192.168.1.100/24"),
      environment_type      => "test",
      environment_host      => "atlas";
      # default environment_state => "create" on host, and "suspended elsewhere"
}

body environment_resources virt_xml(specfile)
{
      env_spec_file => "$(specfile)";
}

body environment_interface vnet(primary)
{
      env_name      => "$(this.promiser)";
      env_addresses => { "$(primary)" };
    host1::
      env_network => "default_vnet1";
    host2::
      env_network => "default_vnet2";
}

Environment variables

code
body common control
{
      bundlesequence  => { "my_vm_cloud" };
}

bundle agent my_vm_cloud
{
  environments:
      "centos5"
      environment_resources => virt_xml,
      environment_type      => "xen",
      environment_host      => "ursa-minor";
      # default environment_state => "create" on host, and "suspended elsewhere"
}

body environment_resources virt_xml
{
      env_spec_file => "/srv/xen/centos5-libvirt-create.xml";
}

Tidying garbage files

Emulating the tidy feature of CFEngine 2.

code
body common control
{
    any::
      bundlesequence  => { "testbundle" };
}

bundle agent testbundle
{
  files:
      "/tmp/test"
      delete => tidy,
      file_select => zero_age,
      depth_search => recurse("inf");
}

body depth_search recurse(d)
{
      #include_basedir => "true";
      depth => "$(d)";
}

body delete tidy
{
      dirlinks => "delete";
      rmdirs   => "false";
}

body file_select zero_age
{
      mtime     => irange(ago(1,0,0,0,0,0),now);
      file_result => "mtime";
}