JSON and YAML Support in CFEngine
Introduction
JSON is a well-known data language. It even has a specification (See http://json.org).
YAML is another well-known data language. It has a longer, much more complex specification (See http://yaml.org).
CFEngine has core support for JSON and YAML. Let's see what it can do.
Problem statement
We'd like to read, access, and merge JSON-sourced data structures: they should be weakly typed, arbitrarily nested, with consistent quoting and syntax.
We'd like to read, access, and merge YAML-sourced data structures just like JSON-sourced, to keep policy and internals simple.
In addition, we must not break backward compatibility with CFEngine
3.5 and older, so we'd like to use the standard CFEngine array a[b]
syntax.
Data containers
A new data type, the data container, was introduced in 3.6.
It's simply called data
. The documentation with some examples is at https://cfengine.com/docs/master/reference-promise-types-vars.html#data-container-variables
Reading JSON
There are many ways to read JSON data; here are a few:
readjson()
: read from a JSON file, e.g."mydata" data => readjson("/my/file", 100k);
parsejson()
: read from a JSON string, e.g."mydata" data => parsejson('{ "x": "y" }');
data_readstringarray()
anddata_readstringarrayidx()
: read text data from a file, split it on a delimiter, and make them into structured data.mergedata()
: merge data containers, slists, and classic CFEngine arrays, e.g."mydata" data => mergedata(container1, slist2, array3);
mergedata
in particular is very powerful. It can convert a slist or a classic CFEngine array to a data container easily: "mydata" data => mergedata(myslist);
Reading YAML
There are two ways to read YAML data:
readyaml()
: read from a YAML file, e.g."mydata" data => readyaml("/my/file.yaml", 100k);
parseyaml()
: read from a YAML string, e.g."mydata" data => parseyaml('- arrayentry1');
Since these functions return data containers, everything about JSON-sourced data structures applies to YAML-sourced data structures as well.
Accessing JSON
To access JSON data, you can use:
- the
nth()
function to access an array element, e.g."myx" string => nth(container1, 0);
- the
nth
function to access a map element, e.g."myx" string => nth(container1, "x");
- the
a[b]
notation, e.g."myx" string => "$(container1[x])";
. You can nest, e.g.a[b][c][0][d]
. This only works if the element is something that can be expanded in a string. So a number or a string work. A list of strings or numbers works. A key-value map underx
won't work. - the
getindices()
andgetvalues()
functions, just like classic CFEngine arrays
A full example
This example can be saved and run. It will load a key-value map where the keys are class names and the values are hostname regular expressions or class names.
- if your host name is
c
orb
or the classesc
orb
are defined, thedev
class will be defined - if your host name is
flea
or the classflea
is defined, theprod
class will be defined - if your host name is
a
or the classa
is defined, theqa
class will be defined - if your host name is
linux
or the classlinux
is defined, theprivate
class will be defined
Easy, right?
body common control
{
bundlesequence => { "run" };
}
bundle agent run
{
vars:
"bykey" data => parsejson('{ "dev": ["c", "b"], "prod": ["flea"], "qa": ["a"], "private": ["linux"] }');
"keys" slist => getindices("bykey");
classes:
# define the class from the key name if any of the items under the key match the host name
"$(keys)" expression => regcmp("$(bykey[$(keys)])", $(sys.host));
# define the class from the key name if any of the items under the key are a defined class
"$(keys)" expression => classmatch("$(bykey[$(keys)])");
reports:
"keys = $(keys)";
"I am in class $(keys)" ifvarclass => $(keys);
}
So, where's the magic? Well, if you're familiar with classic CFEngine
arrays, you will be happy to hear that the exact same syntax works
with them. In other words, data containers don't change how you use
CFEngine. You still use getindices
to get the keys, then iterate
through them and look up values.
Well, you can change
"bykey" data => parsejson('{ "dev": ["c", "b"], "prod": ["flea"], "qa": ["a"], "private": ["linux"] }');
with
"bykey" data => data_readstringarray(...);
and read the same container from a text file. The file should be formatted like this to produce the same data as above:
dev c b
prod flea
qa a
private linux
You can also use
"bykey" data => readjson(...);
and read the same container from a JSON file.
Summary
Using JSON and YAML from CFEngine is easy and does not change how you use CFEngine. Try it out and see for yourself!