Table of Contents

The Design Center API

Table of Contents

API General Information

The Design Center API (DC API or just API henceforth) is a simple JSON-based protocol for communicating with the Design Center backend. The backend may be running locally or remotely; the API makes no assumptions about the transport channel and is entirely a line-based text protocol consisting of one JSON line each way.

The API client makes a request and gets a response over the same channel. Again, the request and the response can only be a single line of text, ended by the transport channel's standard line-ending sequence, e.g. CRLF for HTTP. JSON escapes such sequences so they should not happen anywhere in the payloads.

API requests have the following general structure:

{ dc_api_version: "3.6.0", request: { ...commands... } }

The version is strictly semantically versioned as major.minor.patch. It must match exactly, so you can't have a 3.6.0 client talking to a 3.6.1 server for instance (the client has to say "3.6.1" to be usable). We expect backward compatibility, this is just a way to avoid misunderstandings.

It's possible that CFEngine 3.7.x will keep using the 3.6.0 API, for instance. Think of the API version as the minimum CFEngine version required to use it.

NOTE: Generally, only one command may be specified per request.

API responses look like this:

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "log": [],
        "tags": {},
        "data": {...response data...
        }
    }
}

The top key can be one of the following:

api_ok: the command was processed correctly and the response is enclosed as valid JSON (note that this doesn't mean the response indicates success!!!)
api_error: the command was not processed correctly and the response may not be valid JSON at all. It may be good JSON and even contain keys like api_ok promises, e.g. warnings or success, but you can't rely on that.

The API client may wish to replace unparseable data with {api_error: "BAD JSON (escaped data here)"} or something similar to make the response handler simpler.

Inside the API response, under the api_ok key, you can expect to find the following:

success: indicates, generally speaking, that the command succeeded or failed. Any complex commands can fail in subtle ways, but the API will do its best to make this a good indicator.
errors and warnings: lists of strings that log errors and warnings for the command.
error_tags: key-value array of tag strings assigned to the error messages. This lets the client tell what stages or areas of the command triggered the errors.
log: list of general message strings. This is optional and purely informational.
tags: key-value array of tag strings assigned to the response, not associated with errors. This lets the client tell what stages or areas of the command triggered messages or warnings, or more generally what stages or areas of the command were executed. This is optional and purely informational.
data: the meat of the response plate, if you will. This key contains all the response data that the API command generated. Each command has different return data so the specifics are listed per command.

API Commands

The API commands and their data responses are listed below. Generally they are exclusive of each other, and the order below is the order in which they are answered. Thus, for instance, a request that issues both list and search will get just the list results.

Many commands take terms. Terms are one of the following:

a string (matches any field)
a list of strings (any of them may match any field)
a list of lists, with each one in the following format: either [FIELD, "matches", REGEX] or [FIELD, "equals", STRING] or [[FIELD1, FIELD2,...], "matches", STRING].

`list`

The list command lists installed sketches.

Here are examples of three list commands. The first one lists everything installed.

{ dc_api_version: "3.6.0", request: {list: true } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "list": {
                "/home/tzz/.cfagent/inputs/sketches": {
                    "CFEngine::dclib::3.5.0": "CFEngine::dclib::3.5.0",
                    "CFEngine::dclib": "CFEngine::dclib",
                    "CFEngine::sketch_template": "CFEngine::sketch_template",
                    "VCS::vcs_mirror": "VCS::vcs_mirror",
                    "Security::SSH": "Security::SSH",
                    "Utilities::ping_report": "Utilities::ping_report",
                    "Monitoring::SNMP::Walk": "Monitoring::SNMP::Walk",
                    "Data::Classes": "Data::Classes",
                    "CFEngine::stdlib": "CFEngine::stdlib",
                    "Utilities::ipverify": "Utilities::ipverify"
                },
            "count": 10
            }
        },
        "log": [],
        "tags": {}
    }
}

Note the top-level key under data/list is the name of the repository, which is always a local directory.

The next one takes terms and lists all the sketches whose name satisfies the terms.

{ dc_api_version: "3.6.0", request: {list: [["name", "matches", "(Cloud|CFEngine|Security)"]] } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "list": {
                "/home/tzz/.cfagent/inputs/sketches": {
                    "Security::SSH": "Security::SSH",
                    "CFEngine::dclib::3.5.0": "CFEngine::dclib::3.5.0",
                    "CFEngine::dclib": "CFEngine::dclib",
                    "CFEngine::sketch_template": "CFEngine::sketch_template",
                    "CFEngine::stdlib": "CFEngine::stdlib"
                }
            },
            "count": 5
        },
        "log": [],
        "tags": {}
    }
}

option: `count_only`

When count_only is given as a top-level option with a value of true, only the count is returned..

option: `describe`

When describe is given as a top-level option with a value of true, as in the example below, the returned data is the contents of sketch.json.

{ dc_api_version: "3.6.0", request: {describe: true, list: [["name", "matches", "ping"]] } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "list": {
                "/home/tzz/.cfagent/inputs/sketches": {
                    "Utilities::ping_report": {
                        "namespace": "cfdc_ping",
                        "manifest": {
                            "changelog": {
                                "comment": "changelog"
                            },
                            "test.cf": {
                                "comment": "Test Policy"
                            },
                            "README.md": {
                                "documentation": true
                            },
                            "params/example.json": {
                                "comment": "Example parameters to report on a few hosts connectivity."
                            },
                            "main.cf": {
                                "desc": "main file"
                            }
                        },
                        "interface": ["main.cf"],
                        "metadata": {
                            "authors": ["Nick Anderson <nick@cmdln.org>", "Ted Zlatanov <tzz@lifelogs.com>"],
                            "version": 1.2,
                            "name": "Utilities::ping_report",
                            "license": "MIT",
                            "description": "Report on pingability of hosts",
                            "tags": ["cfdc"],
                            "depends": {
                                "cfengine": {
                                    "version": "3.4.0"
                                },
                                "CFEngine::dclib": {},
                                "os": ["linux"],
                                "CFEngine::stdlib": {
                                    "version": 105
                                }
                            }
                        },
                        "entry_point": null,
                        "api": {
                            "ping": [{
                                "name": "runenv",
                                "type": "environment"
                            },
                            {
                                "name": "metadata",
                                "type": "metadata"
                            },
                            {
                                "name": "hosts",
                                "type": "list"
                            },
                            {
                                "name": "count",
                                "type": "string"
                            },
                            {
                                "name": "reached",
                                "type": "return"
                            },
                            {
                                "name": "not_reached",
                                "type": "return"
                            }]
                        }
                    }
                }
            }
        },
        "log": [],
        "tags": {}
    }
}

When describe is given as a top-level option with a value of README, as in the example below, the returned data is actually the sketch's auto-generated README.md file (which comes from sketch.json). The tools/test/Makefile testing Makefile has a convenience regenerate_readme target to do this for all the DC sketches.

If the manifest includes a README.include file, it will be included verbatim in the README.md in the Description section. That makes it easier to write documentation for your sketches.

{ dc_api_version: "3.6.0", request: {describe: "README", list: [["name", "matches", "ping"]] } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "list": {
                "/home/tzz/.cfagent/inputs/sketches": {
                    "Utilities::ping_report": ["/home/tzz/.cfagent/inputs/sketches/utilities/ping_report", "# Utilities::ping_report version 1.2\n\nLicense: MIT\nTags: cfdc\nAuthors: Nick Anderson <nick@cmdln.org>, Ted Zlatanov <tzz@lifelogs.com>\n\n## Description\nReport on pingability of hosts\n\n## Dependencies\nCFEngine::dclib, CFEngine::stdlib\n\n## API\n### bundle: ping\n* parameter _environment_ *runenv* (default: none, description: none)\n\n* parameter _metadata_ *metadata* (default: none, description: none)\n\n* parameter _list_ *hosts* (default: none, description: none)\n\n* parameter _string_ *count* (default: none, description: none)\n\n* returns _return_ *reached* (default: none, description: none)\n\n* returns _return_ *not_reached* (default: none, description: none)\n\n\n## SAMPLE USAGE\nSee `test.cf` or the example parameters provided\n\n"]
                }
            },
            "count": 1
        },
        "log": [],
        "tags": {}
    }
}

`search`

The search command works exactly like list above, except that the candidate list contains all available sketches (from recognized_sources), not just the installed sketches.

option: `count_only`

When count_only is given as a top-level option with a value of true, only the count is returned..

option: `describe`

The describe option to search works exactly like it does for list above.

`describe`

The describe command gives the contents of sketch.json for the matching installed sketches by name.

{ dc_api_version: "3.6.0", request: {describe:"Security::SSH"} }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "describe": {
                "/home/tzz/.cfagent/inputs/sketches": {
                    "Security::SSH": [{
                        "namespace": "cfdc_sshd",
                        "manifest": {
                            "ssh.cf": {
                                "desc": "main file"
                            },
                            "README.md": {
                                "documentation": true
                            },
                            "params/simple.json": {}
                        },
                        "interface": ["ssh.cf"],
                        "metadata": {
                            "authors": ["Diego Zamboni <diego.zamboni@cfengine.com>", "Ted Zlatanov <tzz@lifelogs.com>"],
                            "version": 1.1,
                            "name": "Security::SSH",
                            "license": "MIT",
                            "description": "Configure and enable sshd",
                            "tags": ["cfdc"],
                            "depends": {
                                "cfengine": {
                                    "version": "3.4.0"
                                },
                                "CFEngine::dclib": {
                                    "version": "1.0.0"
                                },
                                "CFEngine::stdlib": {
                                    "version": 105
                                }
                            }
                        },
                        "api": {
                            "sshd": [{
                                "name": "runenv",
                                "type": "environment"
                            },
                            {
                                "name": "metadata",
                                "type": "metadata"
                            },
                            {
                                "name": "params",
                                "type": "array"
                            }]
                        }
                    }]
                },
                "/home/tzz/source/design-center/sketches": {
                    "Security::SSH": [{
                        "namespace": "cfdc_sshd",
                        "manifest": {
                            "ssh.cf": {
                                "desc": "main file"
                            },
                            "README.md": {
                                "documentation": true
                            },
                            "params/simple.json": {}
                        },
                        "interface": ["ssh.cf"],
                        "metadata": {
                            "authors": ["Diego Zamboni <diego.zamboni@cfengine.com>", "Ted Zlatanov <tzz@lifelogs.com>"],
                            "version": 1.1,
                            "name": "Security::SSH",
                            "license": "MIT",
                            "description": "Configure and enable sshd",
                            "tags": ["cfdc"],
                            "depends": {
                                "cfengine": {
                                    "version": "3.4.0"
                                },
                                "CFEngine::dclib": {
                                    "version": "1.0.0"
                                },
                                "CFEngine::stdlib": {
                                    "version": 105
                                }
                            }
                        },
                        "api": {
                            "sshd": [{
                                "name": "runenv",
                                "type": "environment"
                            },
                            {
                                "name": "metadata",
                                "type": "metadata"
                            },
                            {
                                "name": "params",
                                "type": "array"
                            }]
                        }
                    }]
                }
            }
        },
        "log": [],
        "tags": {}
    }
}

`install`

The install command installs any number of sketches. The data provides is a list of key-value arrays with keys:

force: boolean, false by default. Whether any existing installations of the sketch should be respected or overwritten. Also asks the API to ignore OS and CFEngine version dependencies.
sketch: the sketch name.
target: the sketch install directory. Must be in the API's repolist. Optional; when not given, the first element of the repolist will be used.
source: the sketch source repository. Must be in the API's recognized_sources. Optional; when not given, every element of the recognized_sources will be tried. Can be a string or an array of strings.

{
    dc_api_version: "3.6.0",
    request: {
        install: [{
            sketch: "CFEngine::sketch_template",
            force: true,
        },
        {
            sketch: "VCS::vcs_mirror",
            force: true,
            target: "~/.cfagent/inputs/sketches",
            source: "/home/tzz/source/design-center/tools/test/../../sketches"
        }]
    }
}

The return data is a key-value array as follows, describing the installation details.

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "VCS::vcs_mirror": {
                "params/thrift-lib-perl.json": "/home/tzz/.cfagent/inputs/sketches/utilities/vcs_mirror/params/thrift-lib-perl.json",
                "README.md": "/home/tzz/.cfagent/inputs/sketches/utilities/vcs_mirror/README.md",
                "params/cfengine-core.json": "/home/tzz/.cfagent/inputs/sketches/utilities/vcs_mirror/params/cfengine-core.json",
                "params/cfengine-copbl.json": "/home/tzz/.cfagent/inputs/sketches/utilities/vcs_mirror/params/cfengine-copbl.json",
                "main.cf": "/home/tzz/.cfagent/inputs/sketches/utilities/vcs_mirror/main.cf",
                "params/cfengine-core-runas-tzz.json": "/home/tzz/.cfagent/inputs/sketches/utilities/vcs_mirror/params/cfengine-core-runas-tzz.json"
            },
            "install": {
                "~/.cfagent/inputs/sketches": {
                    "VCS::vcs_mirror": 1,
                    "CFEngine::sketch_template": 1
                }
            },
            "inventory_save": 1,
            "CFEngine::sketch_template": {
                "test.cf": "/home/tzz/.cfagent/inputs/sketches/sketch_template/test.cf",
                "scripts/sample.sh": "/home/tzz/.cfagent/inputs/sketches/sketch_template/scripts/sample.sh",
                "params/demo.json": "/home/tzz/.cfagent/inputs/sketches/sketch_template/params/demo.json",
                "README.md": "/home/tzz/.cfagent/inputs/sketches/sketch_template/README.md",
                "modules/mymodule": "/home/tzz/.cfagent/inputs/sketches/sketch_template/modules/mymodule",
                "main.cf": "/home/tzz/.cfagent/inputs/sketches/sketch_template/main.cf"
            }
        },
        "log": [],
        "tags": {
            "VCS::vcs_mirror": 1,
            "installation": 7,
            "CFEngine::sketch_template": 1
        }
    }
}

`uninstall`

The uninstall command simply deletes the top-level sketch directory and everything under it. It takes a list of key-value arrays with keys:

sketch: the sketch name.
target: the sketch install directory we want to clean. Must be in the API's repolist.

{ dc_api_version: "3.6.0", request: {uninstall: [ { sketch: "CFEngine::stdlib", target: "~/.cfagent/inputs/sketches" } ] } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "inventory_save": 1,
            "uninstall": {
                "~/.cfagent/inputs/sketches": {
                    "CFEngine::stdlib": 1
                }
            }
        },
        "log": [],
        "tags": {
            "uninstallation": 1,
            "CFEngine::stdlib": 1
        }
    }
}

The inventory_save key in the return indicates whether the inventory (cfsketches.json) was written successfully.

`compositions`

The compositions command lists the defined compositions.

{ dc_api_version: "3.6.0", request: {compositions: true} }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "compositions": {
                "mirror_to_template_2": {
                    "destination_sketch": "CFEngine::sketch_template",
                    "source_scalar": "deploy_path",
                    "source_sketch": "VCS::vcs_mirror",
                    "destination_scalar": "myip"
                },
                "mirror_to_template_1": {
                    "destination_sketch": "CFEngine::sketch_template",
                    "source_scalar": "deploy_path",
                    "source_sketch": "VCS::vcs_mirror",
                    "destination_list": "mylist"
                }
            }
        },
        "log": [],
        "tags": {}
    }
}

`compose`

The compose command defines a composition. It returns the same data as compositions.

{
    dc_api_version: "3.6.0",
    request: {
        compose: {
            mirror_to_template_1: {
                destination_sketch: "CFEngine::sketch_template",
                destination_list: "mylist",
                source_sketch: "VCS::vcs_mirror",
                source_scalar: "deploy_path"
            },
            mirror_to_template_2: {
                destination_sketch: "CFEngine::sketch_template",
                destination_scalar: "myip",
                source_sketch: "VCS::vcs_mirror",
                source_scalar: "deploy_path"
            }
        }
    }
}

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "compositions": {
                "mirror_to_template_2": {
                    "destination_sketch": "CFEngine::sketch_template",
                    "source_scalar": "deploy_path",
                    "source_sketch": "VCS::vcs_mirror",
                    "destination_scalar": "myip"
                },
                "mirror_to_template_1": {
                    "destination_sketch": "CFEngine::sketch_template",
                    "source_scalar": "deploy_path",
                    "source_sketch": "VCS::vcs_mirror",
                    "destination_list": "mylist"
                }
            }
        },
        "log": [],
        "tags": {
            "compose": 1
        }
    }
}

`decompose`

The decompose command undefines a composition by name. It returns the same data as compositions.

{ dc_api_version: "3.6.0", request: {decompose: "mirror_to_template_1" } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "compositions": {
                "destination_sketch": "CFEngine::sketch_template",
                "source_scalar": "deploy_path",
                "source_sketch": "VCS::vcs_mirror",
                "destination_list": "mylist"
            }
        },
        "log": [],
        "tags": {
            "compose": 1
        }
    }
}

(Note that Monty Python has beaten us to this joke by decades with "The Decomposing Composers.")

`activations`

The activations command lists the defined activations.

{ dc_api_version: "3.6.0", request: {activations:true} }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "activations": {
                "VCS::vcs_mirror": [{
                    "params": ["vcs_base", "git_mirror_core"],
                    "environment": "testing",
                    "target": "~/.cfagent/inputs/sketches"
                },
                {
                    "params": ["vcs_base", "svn_mirror_thrift"],
                    "environment": "testing",
                    "target": "~/.cfagent/inputs/sketches"
                }],
                "CFEngine::sketch_template": [{
                    "params": ["incomplete_sketch"],
                    "environment": "testing",
                    "target": "~/.cfagent/inputs/sketches",
                    "compositions": ["mirror_to_template_1", "mirror_to_template_2"]
                }]
            }
        },
        "log": [],
        "tags": {}
    }
}

Under each activation you may find an optional hash key identifying it uniquely (it's a hash of the resolved parameters, bundle and sketch name, and run environment name), but this should never be considered mandatory.

`activate`

The activate command defines a new activation of a sketch.

An activation is a matching of a sketch bundle with parameters, a run environment, and optionally compositions. The sketch name is matched with a target (so the API knows which installed sketch to inspect), a run environment name, and a list of parameter names.

{ dc_api_version: "3.6.0", request: {activate: { "VCS::vcs_mirror": { target: "~/.cfagent/inputs/sketches", environment: "testing", params: [ "vcs_base", "git_mirror_core" ] } } } }

The sketch bundle will be selected based on which one is satisfied by the given parameters and compositions. You can use the __bundle__ parameter key to specify the bundle explicitly.

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "activate": {
                "VCS::vcs_mirror": {
                    "params": ["vcs_base", "git_mirror_core"],
                    "environment": "testing",
                    "target": "~/.cfagent/inputs/sketches"
                }
            }
        },
        "log": [],
        "tags": {
            "VCS::vcs_mirror": 1
        }
    }
}

You can pass a identifier parameter to an activate command, which can then be used to deactivate an activation specifically, and which will show up in the classes and prefixes of that activation.

You can pass a priority parameter to an activate command, which will be used for sorting the activations. By default all activations get priority 1. The priorities are sorted lexicographically (00a comes before 00b and 10 comes before 9).

You can pass a metadata parameter to an activate command, which will show up under the activation key in the metadata.

You can pass a target parameter to an activate command with an install location, which will only activate sketches that exist in that location.

You may find a hash key in the result, as described in the activations command above.

option: `compose`

When the activate command has a compose key with a list of composition names, those compositions are considered whenever the parameters alone are not enough to activate the sketch. Thus compositions and parameters work together, as late and immediate bindings of the passed data respectively.

`deactivate`

The deactivate command removes sketch activations. It can take either the name of a sketch or true to indicate all activations should be removed.

{ dc_api_version: "3.6.0", request: {deactivate: "VCS::vcs_mirror" } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "deactivate": {
                "VCS::vcs_mirror": 1
            }
        },
        "log": [],
        "tags": {
            "deactivate": 1
        }
    }
}

{ dc_api_version: "3.6.0", request: {deactivate: true } }

(No activations existed at this point, so the return data is empty.)

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {},
        "log": [],
        "tags": {}
    }
}

`definitions`

The definitions command lists the parameter definitions. This is the DC API's central library of knowledge. Every parameter definition is a source of configuration data (like a CFEngine common bundle, but applied directly to a sketch bundle). Parameter definitions have names, which are used when you want to activate a sketch, and can contain more than one sketch's parameters or only part of a sketch's parameters.

{ dc_api_version: "3.6.0", request: {definitions:true} }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "definitions": {
                "simple_ssh": {
                    "Security::SSH": {
                        "params": {
                            "X11Forwarding": "yes",
                            "Protocol": "2",
                            "PermitRootLogin": "yes"
                        }
                    }
                },
            }
        },
        "log": [],
        "tags": {}
    }
}

`define`

The define command creates a parameter definition with a name. The example here creates some base parameters for the VCS::vcs_mirror sketch and then lays specific configuration to mirror the [https://github.com/cfengine/core.git] repository's master branch from Git. In this case, we do it in two steps, but could have done it in one step.

Note that the reply doesn't tell you more than "I got it, thanks."

You can use the function expression in data, as shown below, to make sure that the DC API will make a function call and not just pass a string. So, instead of getenv("LOGNAME", "128") you need to use { "function": "getenv", "args": ["LOGNAME", "128"] } to make sure the function call is preserved.

{ dc_api_version: "3.6.0", request: {define: { "vcs_base": { "VCS::vcs_mirror": { options: { parent_dir: { owner: { "function": "getenv", "args": ["LOGNAME", "128"] }, group: { "function": "getenv", "args": ["LOGNAME", "128"] }, perms: "755", ensure: true }, nowipe: true, vcs: { runas: { "function": "getenv", "args": ["LOGNAME", "128"] }, umask: "000" } } } } } } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "define": {
                "vcs_base": 1
            }
        },
        "log": [],
        "tags": {
            "vcs_base": 1
        }
    }
}

{ dc_api_version: "3.6.0", request: {define: { "git_mirror_core": { "VCS::vcs_mirror": { vcs: "/usr/bin/git", path: "/tmp/q/cfengine-core", branch: "master", origin: "https://github.com/cfengine/core.git" } } } } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "define": {
                "git_mirror_core": 1
            }
        },
        "log": [],
        "tags": {
            "git_mirror_core": 1
        }
    }
}

`undefine`

The undefine command removes a parameter definition by name. You can pass a list of string parameter definition names or simply true to remove all the parameter definitions.

{ dc_api_version: "3.6.0", request: {undefine: ["git_mirror_core"] } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "undefine": {
                "git_mirror_core": "1"
            }
        },
        "log": [],
        "tags": {
            "git_mirror_core": 1
        }
    }
}

`environments`

The environments command lists the run environments.

A run environment is a common bundle of general settings. It affects the execution of bundles globally, so it's not intended to be specific for each bundle activation.

The sketch bundle chooses to have a run environment by specifying a parameter with type environment. Only a run environment can satisfy that API parameter.

Good examples of run environments are production, production_debug, or development_nodebug. In a run environment you'd expect to find at least the activated, verbose, and test variables. For each of those, the DC API will also provide a class named runenv_ENVIRONMENTNAME_ENVIRONMENTVARIABLE. Here's an example of a testing run environment, as it appears in the generated runfile:

bundle common testing
{
  vars:
      "activated" string => "1";
      "env_vars" slist => { "activated", "test", "verbose" };
      "test" string => "1";
      "verbose" string => "1";
  classes:
      "runenv_testing_activated" expression => "any";
      "runenv_testing_test" expression => "any";
      "runenv_testing_verbose" expression => "any";
}

And here is the definition of that run environment:

{ dc_api_version: "3.6.0", request: {environments:true} }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "environments": {
                "testing": {
                    "verbose": "1",
                    "test": "1",
                    "activated": "1"
                }
            }
        },
        "log": [],
        "tags": {}
    }
}

The last thing to note is that any run environment variable can have values other than true and false. If they are a string, then that string is a class expression. So, for instance, if activated is Monday then the run environment will only be activated on Mondays.

It's trivial to do AND and OR in such a string, as normal for CFEngine contexts.

`define_environment`

The define_environemnt command defines a run environment. The testing example above can be defined like so:

{ dc_api_version: "3.6.0", request: {define_environment: { "testing": { activated: true, test: true, verbose: true } } } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "define_environment": {
                "testing": 1
            }
        },
        "log": [],
        "tags": {
            "testing": 1
        }
    }
}

Again, remember that each of those variables can be a string, to be interpreted as a class expression, and that you can have more than those three variables.

`undefine_environment`

The undefine_environemnt command removes a run environment. It takes a list of environment names.

{ dc_api_version: "3.6.0", request: {undefine_environment: [ "testing" ] } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "undefine_environment": {
                "testing": "1"
            }
        },
        "log": [],
        "tags": {
            "testing": 1
        }
    }
}

`validations`

The validations command lists the data validations.

The data validations are just strings that have a key-value array associated with them. Specific keys trigger specific validation behavior in order, as follows. Note that the examples below are not necessarily in your API installation already.

// only the inside of the request is shown for brevity
define_validation: { DIGITS: { valid_regex: "^[0-9]+$" } }
define_validation: { NUMBER: { derived: [ "DIGITS" ] } }
define_validation: { AB: { choice: [ "A", "B" ] } }
define_validation: { 8BIT_NUMBER: { minimum_value: 0, maximum_value: 255 } }
define_validation: { LIST_OF_NUMBERS: { list: [ "NUMBER" ] } }
define_validation: { MOG_SEQUENCE: { sequence: [ "OCTAL", "UID", "GID"   ] } }
define_validation: { ARRAY_OF_NUMBERS_TO_URLS: { array_k: [ "NUMBER" ], array_v: [ "URL" ] } }

derived defines a parent data validation. So a NUMBER validation requires that DIGITS and any other parent data validations be checked first.
choice defines a list of exact string matches. So AB must be given A or B to pass validation.
minimum_value and then maximum_value are numeric checks. So 8BIT_NUMBER has to be between 0 and 255. Any invalid numbers, e.g. hello, will be treated as 0.
invalid_regex and then valid_regex are regular expressions written as strings. They follow the Perl regex syntax right now. So DIGITS can only contain the decimal digits 0 through 9 and will reject the empty string `orhello`.
invalid_ipv4 and valid_ipv4 are TODO.
list ensures that the given data is a list of one of several data types. So in the example, LIST_OF_NUMBERS will check that every element passes the NUMBER validation.
sequence is like a record: it ensures that the data is a sequence (list) of the given data types. So for example, MOG_SEQUENCE has to have three elements, of which the first one passes OCTAL validation, the second passed UID validation, and the third passes GID validation.
array_k and array_v are almost exactly like list but they validate the keys and values of a key-value array, respectively, against a list of several data types. So ARRAY_OF_NUMBERS_TO_URLS requires that every key pass the NUMBER validation and every value pass the URL validation.

`define_validation`

The define_validation command defines a data validation. In the return data you will find all the currently defined data validations.

{ dc_api_version: "3.6.0", request: {define_validation: { NONEMPTY_STRING: { valid_regex: "." } } } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "validations": {
                "NONEMPTY_STRING": {
                    "valid_regex": "."
                },
            }
        },
        "log": [],
        "tags": {
            "define_validation": 1
        }
    }
}

`undefine_validation`

The undefine_validation command removes a data validation by name.

{ dc_api_version: "3.6.0", request: {undefine_validation: "NONEMPTY_STRING" } }'

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "validations": {
                "valid_regex": "."
            }
        },
        "log": [],
        "tags": {
            "undefine_validation": 1
        }
    }
}

`validate`

The validate command validates data using a named data validation.

{ dc_api_version: "3.6.0", request: {validate: { validation: "ARRAY_OF_NUMBERS_TO_URLS", data: { "20": "http://this.that", "30": "not a URL" } } } }

It's useful to look at the log output here. This example failed:

DCAPI::log4(Validation.pm:73): Validating ARRAY_OF_NUMBERS_TO_URLS against data '{"30":"not a URL","20":"http://this.that"}'
DCAPI::log4(Validation.pm:282): Validating ARRAY_OF_NUMBERS_TO_URLS: checking 'array_k' is ["NUMBER"]
DCAPI::log4(Validation.pm:73): Validating NUMBER against data '30'
DCAPI::log4(Validation.pm:73): Validating DIGITS against data '30'
DCAPI::log4(Validation.pm:166): Validating DIGITS: checking valid_regex ^[0-9]+$
DCAPI::log4(Validation.pm:85): Validating NUMBER: checking parent data type DIGITS
DCAPI::log4(Validation.pm:73): Validating NUMBER against data '20'
DCAPI::log4(Validation.pm:73): Validating DIGITS against data '20'
DCAPI::log4(Validation.pm:166): Validating DIGITS: checking valid_regex ^[0-9]+$
DCAPI::log4(Validation.pm:85): Validating NUMBER: checking parent data type DIGITS
DCAPI::log4(Validation.pm:282): Validating ARRAY_OF_NUMBERS_TO_URLS: checking 'array_v' is ["URL"]
DCAPI::log4(Validation.pm:73): Validating URL against data 'not a URL'
DCAPI::log4(Validation.pm:166): Validating URL: checking valid_regex ^[A-Za-z]{3,9}://.+
DCAPI::log4(Validation.pm:73): Validating URL against data 'http://this.that'
DCAPI::log4(Validation.pm:166): Validating URL: checking valid_regex ^[A-Za-z]{3,9}://.+

{
    "api_ok": {
        "warnings": [],
        "success": false,
        "errors": ["Could not validate any of the allowed array_v types [URL]"],
        "error_tags": {
            "array_v": 1,
            "validation": 1
        },
        "data": {},
        "log": [],
        "tags": {}
    }
}

This example succeeded:

{ dc_api_version: "3.6.0", request: {validate: { validation: "ARRAY_OF_NUMBERS_TO_URLS", data: { "20": "http://this.that", "30": "http://this.that2" } } } }

DCAPI::log4(Validation.pm:73): Validating ARRAY_OF_NUMBERS_TO_URLS against data '{"30":"http://this.that2","20":"http://this.that"}'
DCAPI::log4(Validation.pm:282): Validating ARRAY_OF_NUMBERS_TO_URLS: checking 'array_k' is ["NUMBER"]
DCAPI::log4(Validation.pm:73): Validating NUMBER against data '30'
DCAPI::log4(Validation.pm:73): Validating DIGITS against data '30'
DCAPI::log4(Validation.pm:166): Validating DIGITS: checking valid_regex ^[0-9]+$
DCAPI::log4(Validation.pm:85): Validating NUMBER: checking parent data type DIGITS
DCAPI::log4(Validation.pm:73): Validating NUMBER against data '20'
DCAPI::log4(Validation.pm:73): Validating DIGITS against data '20'
DCAPI::log4(Validation.pm:166): Validating DIGITS: checking valid_regex ^[0-9]+$
DCAPI::log4(Validation.pm:85): Validating NUMBER: checking parent data type DIGITS
DCAPI::log4(Validation.pm:282): Validating ARRAY_OF_NUMBERS_TO_URLS: checking 'array_v' is ["URL"]
DCAPI::log4(Validation.pm:73): Validating URL against data 'http://this.that2'
DCAPI::log4(Validation.pm:166): Validating URL: checking valid_regex ^[A-Za-z]{3,9}://.+
DCAPI::log4(Validation.pm:73): Validating URL against data 'http://this.that'
DCAPI::log4(Validation.pm:166): Validating URL: checking valid_regex ^[A-Za-z]{3,9}://.+

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {},
        "log": [],
        "tags": {}
    }
}

`regenerate`

The regenerate command writes the API runfile (as specified in the API configuration) from all the known activations, compositions, run environments, parameter definitions, and data validations.

The command does not allow the user to change the runfile location, as that is a possible security risk.

In the returned data you can find the runfile name and also each activation in a key-value map, indexed by the internal activation unique name, and with each value a list of name, sketch, bundle, parameter checksum. The parameter checksum is calculated from the final parameters after the parameter definitions have been resolved, so compositions will be recognized as well. Here's an example regeneration with just one activation, with the runfile location in a specific user's home directory (this is typically how cf-sketch.pl will configure itself if it runs as non-root).

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "___001_System_motd_entry": ["", "System::motd", "entry", "b3172b7755c090fd49e0b250f6320880"],
            "runfile":"/home/tzz/.cfagent/inputs/sketches/meta/api-runfile.cf"
            },
        "log": [],
        "tags": {
            "activations": 1
        }
    }
}

`regenerate_index`

The regenerate_index command takes a directory parameter (string) and writes the cfsketches.json index from all the sketches found in a given directory. The directory must be local and listed in the API configuration's recognized_sources. The command returns an error if the index could not be written or if an error happened while loading any sketch.json files.

{ dc_api_version: "3.6.0", request: {regenerate_index: "~/source/cfengine/design-center/sketches" } }

DCAPI::log3(DCAPI.pm:1500): Regenerating index: searching for sketches in ~/source/cfengine/design-center/sketches
DCAPI::log3(DCAPI.pm:1523): Regenerating index: on sketch dir applications/memcached
...
DCAPI::log3(DCAPI.pm:1523): Regenerating index: on sketch dir web_servers/apache

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {},
        "log": [],
        "tags": {}
    }

`test`

The test command tests installed sketches. It always returns true if the test harness ran, even if the individual tests failed. It's up to you to check the result of each sketch's test.

Here are examples of two test commands. The first one tests everything installed (shown when no sketches were installed for brevity; see below for a full test example).

{ dc_api_version: "3.6.0", request: {test: true } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "coverage": 0,
            "test": {},
            "total": 0
            }
        },
        "log": [],
        "tags": {}
    }
}

Under data you will find a coverage and a total key, which respectively represent the number of covered sketches and the total number of sketches inspected. So if you asked to test 10 sketches but only one had any test scripts, your coverage would be 1/10.

The top-level key under data.test is the name of the repository, which is always a local directory.

The next one takes terms and tests all the sketches whose name satisfies the terms. The return format is the same: for each repository and each sketch tested, you'll get a key-value array with keys log (the text log of the output); failed (with tests that failed); and total (with all the tests).

The format inside each test is according to the Perl module Test::Harness. For instance the good key will be 1 if all the planned tests succeeded.

The bench key will give you some timings, but more precise timings may be added in the future. Do not depend on the format of the bench value.

{ dc_api_version: "3.6.0", request: {test: ["Applications::Memcached"] } }

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "test": {
                "/home/tzz/.cfagent/inputs/sketches": {
                    "Applications::Memcached": {
                        "log": "/home/tzz/.cfagent/inputs/sketches/applications/memcached/test.pl .. \n1..6\n# Running under perl version 5.014002 for linux\n# Current time local: Tue May  7 18:08:08 2013\n# Current time GMT:   Tue May  7 22:08:08 2013\n# Using Test.pm version 1.25_02\nok 1\nok 2\nok 3\nok 4\nok 5\nok 6\nok\n",
                        "failed": {},
                        "total": {
                            "files": 1,
                            "max": 6,
                            "bonus": 0,
                            "skipped": 0,
                            "sub_skipped": 0,
                            "ok": 6,
                            "bad": 0,
                            "good": 1,
                            "tests": 1,
                            "bench": " 1 wallclock secs ( 0.02 usr  0.00 sys +  0.45 cusr  0.01 csys =  0.48 CPU)",
                            "todo": 0
                        }
                    }
                }
            }
        },
        "log": [],
        "tags": {}
    }
}

You can skip the actual testing and just get the coverage if you give the test command the coverage parameter. Here's how you can inspect the coverage of every single installed sketch:

{"dc_api_version":"3.6.0","request":{"coverage":1,"test":["1"]}}

{
    "api_ok": {
        "warnings": [],
        "success": true,
        "errors": [],
        "error_tags": {},
        "data": {
            "coverage": 7,
            "test": {
                "/home/tzz/.cfagent/inputs/sketches": {
                    "System::Syslog": 0,
                    "Networking::NTP::Client": 0,
// ...
                    "Packages::installed": 1,
                    "CFEngine::dclib::3.5.0": 1,
                }
            },
            "total": 32
        },
        "log": [],
        "tags": {}
    }
}

`selftests`

The selftests command lists the internal API tests. It's for internal use only and so left undocumented.

`selftest`

The selftest command runs internal API tests. It's for internal use only and so left undocumented.

API CLI Interface and config.json

From the command line, you can run cd tools/cf-sketch; ./cf-dc-api.pl config.json where config.json contains the necessary configuration for the API:

`log`

Either STDOUT or STDERR or a file name.

`log_level`

1-5 currently. 4 or 5 for debugging; 1 or 2 for normal usage.

3 is for people who can't make up their mind.

`repolist`

A list of local directories where sketches may be installed.

`recognized_sources`

A list of DC repositories where sketches may be installed FROM. There can be local directories or URLs.

`runfile`

A key-value array.

The key location specifies the place where the runfile is written. If not specified, it defaults to the first element of repolist plus /meta/api-runfile.cf.

The standalone, standalone_inputs, and relocate_path keys were available in older versions but are ignored as of DC API version 3.6.0.

If you specify the key filter_inputs with an array value, any inputs matching any elements in that array will be omitted from the generated runfile. That way you can, for example, exclude the cfengine_stdlib.cf that Design Center provides.

If you specify the string header under runfile, it will be inserted before any other comments (so you can have, for instance, a comment).

`vardata`

The file location where the API will record all data. If not specified, it defaults to the first element of repolist plus /meta/vardata.conf.

You should avoid changing this explicitly without very good reason.

A vardata of - (just a dash) means that the API will not try to write to the vardata file, but will pretend everything is OK.

`constdata`

The file location for the pre-defined API validations and other constant DC data. If not specified, it defaults to the first element of repolist plus /meta/constdata.conf.

You should avoid changing this explicitly without very good reason.

A constdata of - (just a dash) means that the API will not try to load the constdata file, but will pretend everything is OK.

Full `config.json` example

The constdata, vardata, and runfile location are left as the default.

{
 log: "STDERR",
 log_level: 4,
 repolist: [ "~/.cfagent/inputs/sketches" ],
 recognized_sources: [ "~/source/design-center/sketches" ],
 runfile: {
            header: "# This file is maintained by CFEngine",
            filter_inputs: [ "some bad file" ],
          },
}

The Design Center API

API General Information

NOTE: Generally, only one command may be specified per request.

API Commands

list

option: count_only

option: describe

search

option: count_only

option: describe

describe

install

uninstall

compositions

compose

decompose

activations

activate

option: compose

deactivate

definitions

define

undefine

environments

define_environment

undefine_environment

validations

define_validation

undefine_validation

validate

regenerate

regenerate_index

test

selftests

selftest

API CLI Interface and config.json

log

log_level

repolist

recognized_sources

runfile

vardata

constdata

Full config.json example

`list`

option: `count_only`

option: `describe`

`search`

option: `count_only`

option: `describe`

`describe`

`install`

`uninstall`

`compositions`

`compose`

`decompose`

`activations`

`activate`

option: `compose`

`deactivate`

`definitions`

`define`

`undefine`

`environments`

`define_environment`

`undefine_environment`

`validations`

`define_validation`

`undefine_validation`

`validate`

`regenerate`

`regenerate_index`

`test`

`selftests`

`selftest`

`log`

`log_level`

`repolist`

`recognized_sources`

`runfile`

`vardata`

`constdata`

Full `config.json` example