Monitoring is the sampling of system variables at regular intervals in order to present an overview of actual changes taking place over time. Monitoring data are often presented as extensive views of moving-line time series. Monitoring has the ability to detect anomalous behaviour by comparing past and present.
The term reporting is usually taken to mean the creation of short summaries of specific system properties suitable for management. System reports describe both promises about the system, such as compliance, discovered changes and faults.
The challenge of both these activities is to compare intended or promised, behaviour with the actual observed behaviour of the system.
The traditional view of IT operations is that configuration, monitoring and reporting a three different things that should not be joined. Traditionally, all three have been independent centralized processes. This view has emerged historically, but it has a major problem.
Monitoring as an independent activity is inherently non-scalable. When numbers of hosts grow beyond a few thousands, centralized monitoring schemes fail to manage the information. Tying configuration (and therefore repair) to monitoring at the host level is essential for the effective management of large and distributed data facilities. CFEngine foresaw this need in 1998, with its Computer Immunology initiative, and continues to develop this strategy.
Cfengine's approach is to focus on scalability. The commercial editions of CFEngine provide what meaningful information they can in a manner that can be scaled to tens of thousands of machines.
A Configuration Management Database (CMDB), sometimes called a Change and Configuration Management Database (CCMDB), is a repository of information about hardware and its expected state. CMDBs are basically an outgrowth of inventory databases. They were embraced by many IT companies as a plausible solution to configuration management, in the absence of an alternative.
A CMDB records so-called configuration items (CI), which record technical, ownership and proposed state data. A CMDB does not generally record actual state – this is left for monitoring software.
The term CMDB originates from the IT Infrastructure Library (ITIL) framework, where it is given a prominent role in system planning and verification. The CMDB is by definition a centralized repository.
CFEngine deviates from the classical view of a CMDB to modernize the concept for modern scaling requirements. Configuration management requires something more sophisticated than a database to describe system patterns. The CMDB applies a brute force approach to collection and searching of system data that does not scale well, and requires large and expensive resources to manage.
CFEngine works with a highly compressed description of system properties that is based on category classification rather than an exhaustive inventory of computers. Being distributed in operation, CFEngine can also enforce policy on state, without brute force imaging.
CFEngine is designed to scale to vast numbers of machines. It does so because it is fundamentally decentralized. Data about system state are recorded in situ and are not transmitted over the network directly. CFEngine summarizes and compresses system information before making it available for central aggregation and analysis. This means that high resolution data are available where they count, processing is decentralized, and inexpensive summaries may be compared and mined for correlations.
Cfengine's distributed architecture means that no data are lost if the network fails temporarily. CFEngine uses the network opportunistically, but it is not reliant on it for real-time operations.
| If you have regular reporting needs, we recommend using our commercially supported version of CFEngine (CFEngine Nova or above), as you will save considerable time and resources in programming, and you will have access to the latest developments through the software subscription. |
No promises made in CFEngine imply automatic aggregation of data to a central location. In commercial cfengine versions, e.g. CFEngine Nova, an optimized aggregation of standardized reports is provided, but the ultimate decision to aggregate must be yours.
Monitoring and reporting capabilities in CFEngine depend on the software version include:
Cfengine's default behaviour is to report to the console (known as standard output). It's default behaviour is to report nothing except errors that are judged to be of a critical nature.
By using cfengine with the inform flag:
# cf-agent -I # cf-agent --informyou can alter the default to report on action items (actual changes) and warnings.
By using cfengine with the verbose flag:
# cf-agent -v # cf-agent --verboseyou can alter the default to report all of its thought-processes. You should not interpret a message that only appears in cfengine's verbose mode as an actual error, only as information that might be relevant to decisions being made by the agent.
CFEngine allows you to use
reports promises to
make reports of your own. A simple example of this is shown below.
body common control
{
bundlesequence => { "test" };
}
#
bundle agent test
{
reports:
cfengine_3::
"$(sys.date),This is a report"
report_to_file => "/tmp/test_log";
}
We can apply this idea to make more useful custom reports. In this example, the agent tests for certain software package and creates a simple HTML file of existing software.
body common control
{
bundlesequence => { "test" };
}
#
bundle agent test
{
vars:
"software" slist => { "gpg", "zip", "rsync" };
classes:
"no_report" expression => fileexists("/tmp/report.html");
"have_$(software)" expression => fileexists("/usr/bin/$(software)");
reports:
no_report::
"
<html>
Name of this host is: $(sys.host)<br>
Type of this host is: $(sys.os)<br>
"
report_to_file => "/tmp/report.html";
#
"
Host has software $(software)<br>
"
ifvarclass => "have_$(software)",
report_to_file => "/tmp/report.html";
#
"
</html>
"
report_to_file => "/tmp/report.html";
}
The outcome of this promise is a file called /tmp/report.html containing output like this:
<html>
Name of this host is: atlas<br>
Type of this host is: linux<br>
Host has software gpg<br>
Host has software zip<br>
Host has software rsync<br>
</html>
The mechanism shown above, can clearly be used to create a wide variety of report formats, but it requires a lot of coding and maintenance by the user.
CFEngine Nova simplifies this kind of report generation by enabling
and updating many out-of-the-box reports directly from the
cf-report agent.
|
CFEngine generates information internally that you might want to use
in reports. For example, the agent cf-agent
interfaces with the local light-weight monitoring
agent cf-monitord so that system state can be reported simply:
body common control
{
bundlesequence => { "report" };
}
###########################################################
bundle agent report
{
reports:
linux::
"/etc/passwd except $(const.n)"
showstate => { "otherprocs", "rootprocs" };
}
A corollary to this is that you can get cfengine to report system anomalies.
reports:
rootprocs_high_dev2::
"RootProc anomaly high 2 dev on $(mon.host) at approx $(mon.env_time)
measured value $(mon.value_rootprocs)
average $(mon.average_rootprocs) pm $(mon.stddev_rootprocs)"
showstate => { "rootprocs" };
entropy_www_in_high&anomaly_hosts.www_in_high_anomaly::
"High entropy incoming www anomaly on $(mon.host) at $(mon.env_time)
measured value $(mon.value_www_in)
average $(mon.average_www_in) pm $(mon.stddev_www_in)"
showstate => { "incoming.www" };
This produces standard output of the form:
R: State of otherprocs peaked at Tue Dec 1 12:12:21 2009 R: The peak measured state was q = 98: R: Frequency: [kjournald] |** (2/98) R: Frequency: [pdflush] |** (2/98) R: Frequency: /var/cfengine/bin/cf-execd|** (2/98) R: Frequency: COMMAND |* (1/98) R: Frequency: init [5] |* (1/98) R: Frequency: [kthreadd] |* (1/98) R: Frequency: [migration/0] |* (1/98) R: Frequency: [ksoftirqd/0] |* (1/98) R: Frequency: [events/0] |* (1/98) R: Frequency: [khelper] |* (1/98) R: Frequency: [kintegrityd/0] |* (1/98) |
Finally, you can quote lines from files in your data for convenience.
body common control
{
bundlesequence => { "report" };
}
###########################################################
bundle agent report
{
reports:
linux::
"/etc/passwd except $(const.n)"
printfile => pr("/etc/passwd","5");
}
######################################################################
body printfile pr(file,lines)
{
file_to_print => "$(file)";
number_of_lines => "$(lines)";
}
This produces output of the form
R: /etc/passwd except R: at:x:25:25:Batch jobs daemon:/var/spool/atjobs:/bin/bash R: avahi:x:103:105:User for Avahi:/var/run/avahi-daemon:/bin/false R: beagleindex:x:104:106:User for Beagle indexing:/var/cache/beagle:/bin/bash R: bin:x:1:1:bin:/bin:/bin/bash R: daemon:x:2:2:Daemon:/sbin:/bin/bash |
Logs can be attached to any promise. In this example, an executed shell command logs a message to the standard output. CFEngine recognizes the
stdout
filename for Standard Output, in the Unix/C standard manner.
bundle agent test
{
commands:
"/tmp/myjob",
action => logme("executor");
}
############################################
body action logme(x)
{
log_repaired => "stdout";
logstring => " -> Started the $(x) (success)";
}
In this next example, a file creation promise logs different outcomes (success or failure) to different log files.
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
vars:
"software" slist => { "/root/xyz", "/tmp/xyz" };
files:
"$(software)"
create => "true",
action => logme("$(software)");
}
#
body action logme(x)
{
log_kept => "/tmp/private_keptlog.log";
log_failed => "/tmp/private_faillog.log";
log_repaired => "/tmp/private_replog.log";
log_string => "$(sys.date) $(x) promise status";
}
This generates three different logs with outputs in of the form:
atlas$ more /tmp/private_keptlog.log Sun Dec 6 11:58:16 2009 /tmp/xyz promise status Sun Dec 6 11:58:43 2009 /tmp/xyz promise status |
CFEngine interfaces with the system logging tools in different ways. Syslog is the default log for Unix-like systems, while the event logger is the default on Windows. You may choose to copy a fixed level of cfengine's standard screen messaging to the system logger on a per-promise basis.
body common control
{
bundlesequence => { "one" };
}
bundle agent one
{
files:
"/tmp/xyz"
create => "true",
action => log;
}
body action log
{
log_level => "inform";
}
In commercial versions of cfengine, you can extract data from the system in more sophisticated ways from files or pipes, using Perl Compatible Regular Expressions to match text. The
cf-monitord
agent is responsible for processing measurement promises.
In this example, we count lines matching a pattern in a file. You might want to scan a log for instances of a particular message and trace this number over time.
bundle monitor watch
{
measurements:
"/tmp/file"
handle => "line_counter",
stream_type => "file",
data_type => "counter",
match_value => scanlines("MYLINE.*"),
history_type => "log";
}
#
body match_value scanlines(x)
{
select_line_matching => "^$(x)$";
}
See the CFEngine Nova documentation for more possibilities of measurement promises.
Total auditing of a system is a surprisingly difficult thing to do, and it is extremely resource intensive. The followers of an audit trail are often paranoid by nature and are seldom satisfied with the level of detail they find. However, the times we really need an audit are rare, but the cost is ever present. The price of certainty is high.
| Spend a moment considering this: if you want to describe every change of state that happens on a computer, then you need to remember old state and compare it to new state. Then you have to record the differences. So you need more than the entire size of your computer's normal resources to do this. Your storage efficiency will always be less than 50% and your processing efficiency will be less than 50% on every audited item. Is this worth the effort? Perhaps your resources would be better spent keeping targeted backups and simply rebuilding contaminated systems. |
Switch on auditing like this:
body agent control
{
auditing => "true";
}
If you decide to go for full auditing, CFEngine will not collect and centralize the reports as they will be too large for this to be a scalable operation. Still, you can view them in a web browser on the local host, or copy them manually to a suitable location.
Doing a change detection scan is a convergent process, but it can still detect changes and present the data in a compressed format that is often more convenient than auditing. The result is less precise, but there is a trade-off between precision and cost.
To make a change tripwire, you use a files promise, something like this:
body common control
{
bundlesequence => { "testbundle" };
}
#
bundle agent testbundle
{
files:
"/home/mark/tmp" -> "me"
changes => scan_files,
depth_search => recurse("inf");
}
# library code ...
body changes scan_files
{
report_changes => "all";
update_hashes => "true";
}
body depth_search recurse(d)
{
depth => "$(d)";
}
In CFEngine Nova, reports of the following form are generated when these promises are kept by the agent:
Change detected File change Sat Dec 5 18:27:44 2009 group for /tmp/testfile changed 100 -> 0 Sat Dec 5 18:27:44 2009 /tmp/testfile Sat Dec 5 18:20:45 2009 /tmp/testfile |
These reports are generated automatically in CFEngine Nova, and are integrated into the web browsable knowledge map. Community edition users have to extract the data and create these themselves.
Table of Contents