io7m | single-page | multi-page | epub | Certusine User Manual

Certusine User Manual

CREATOR Mark Raynsford
DATE 2022-07-01T12:36:21+00:00
DESCRIPTION Documentation for the Certusine ACME client.
IDENTIFIER 26192579-ab8b-4f62-b825-c98d0bb81d6b
LANGUAGE en
RIGHTS Public Domain
TITLE Certusine User Manual
The certusine client is an ACME client with a focus on minimalism, a small footprint, and reliability. The client has the following notable features:

1.2. Features

  • Uses acme4j internally for strong RFC compliance.
  • Exclusively uses the DNS-01 ACME challenge type for ease of integration with infrastructure without having to set up insecure web servers.
  • A small, easily auditable codebase with a heavy use of modularity for correctness.
  • Exposes a service provider API for integrating with new DNS APIs.
  • Exposes a service provider API for implementing new types of certificate outputs.
  • Supports Hetzner DNS.
  • Supports Vultr DNS.
  • Supports Gandi LiveDNS.
  • Supports writing certificates to looseleaf servers.
  • Heavily instrumented with OpenTelemetry for reliable service monitoring.
  • An extensive automated test suite with high coverage.
  • A small footprint; the client is designed to run in tiny 16-32mb JVM heap configurations.
  • Platform independence. No platform-dependent code is included in any form, and installations can largely be carried between platforms without changes.
  • OSGi-ready
  • JPMS-ready
  • ISC license
The certusine package is available from the following sources:
Regardless of the distribution method, the certusine package will contain a command named certusine that acts as the main entrypoint to all the package's functionality. The certusine command expects an environment variable named CERTUSINE_HOME to be defined that points to the installation directory. See the documentation for the installation methods below for details.
A distribution package can be found at Maven Central.
The certusine command requires that a Java 21+ compatible JVM be accessible via /usr/bin/env java.
Verify the integrity of the distribution zip file:

2.2.4. Verify

$ gpg --verify com.io7m.certusine.cmdline-3.2.0-distribution.zip.asc
gpg: assuming signed data in 'com.io7m.certusine.cmdline-3.2.0-distribution.zip.asc'
gpg: Signature made Tue 28 Jun 2022 15:01:56 GMT
gpg:                using RSA key 3CCE59428B30462D10459909C5607DA146E128B8
gpg:                issuer "contact@io7m.com"
gpg: using pgp trust model
gpg: Good signature from "io7m.com (2022 maven-rsa-key) <contact@io7m.com>" [unknown]
Unzip the zip file, and set CERTUSINE_HOME appropriately:

2.2.6. Extract

$ unzip com.io7m.certusine.cmdline-3.2.0-distribution.zip
$ export CERTUSINE_HOME=$(realpath certusine)
$ ./certusine/bin/certusine
info: Usage: certusine [options] [command] [command options]
...
OCI images are available from Quay.io for use with podman or docker.

2.3.1.2. Podman/Docker

$ podman pull quay.io/io7mcom/certusine:3.2.0
$ podman run quay.io/io7mcom/certusine:3.2.0
certusine: usage: certusine [command] [arguments ...]
...
The certusine package uses SQLite internally. The certusine package is written in Java and includes native SQLite libraries compiled for a wide range of platforms. At run-time, the native library for the current platform is extracted to a temporary directory. By default, the standard temporary directory specified by the java.io.tmpdir system property is used (/tmp on most Unix-like systems).
On systems using SELinux, applications are not allowed to execute code from binaries or libraries that are in a directory that has the tmp_t SELinux type. It is practically guaranteed that any system-wide temporary directory will be configured to have this type. Therefore, in order to run certusine on these systems, it is necessary to specify a directory using the org.sqlite.tmpdir system property. As an example, the OCI image configures this property to use /certusine/var to store the temporary native libraries.
Versions of the certusine package prior to 3.0.0 used the MVStore database to store internal certificate state. In version 3.0.0, the certusine package switched to using SQLite.
Please either delete the existing store before running 3.0.0 on a system that previously used 2.*.*, or specify a new file name for the store in the configuration file. Certificates will be automatically recreated and placed into the new store on application startup.
The certusine client accepts an XML configuration file with a very strictly-defined schema. The configuration file is an XML element with the following elements:
The certusine package exclusively uses ISO 8601 syntax when specifying durations of time. For example, the string PT1M specifies a one-minute duration. The string PT1H specifies a one-hour duration.
The xmlns attribute must be present and set to the value "urn:com.io7m.certusine:configuration:1".
The Options element specifies global options for the client.
The DNSWaitTime attribute specifies the amount of time that the client will wait between creating DNS records, and then notifying the ACME servers that the records have been created. This wait time is necessary because DNS records sometimes take time to propagate, and if the client instructs the ACME server to check the records before they have had time to propagate, then the certificate authorization check will fail.
The CertificateStore attribute specifies the file that the client will use for its internal database of certificates. Relative paths are resolved relative to the configuration file.
The CertificateExpirationThreshold attribute specifies the maximum amount of time before expiration that the client will allow before it attempts to renew a certificate. For example, a value of PT72H means that the client will start attempting to renew a certificate when the certificate becomes due to expire in less than 72 hours.

3.4.5. Example Options

<Options DNSWaitTime="PT5M"
         CertificateStore="store.db"
         CertificateExpirationThreshold="PT72H"/>
The Accounts element specifies a set of ACME accounts. Most installations will only use a single account.
The Name attribute for a given account specifies the name of that account. Names can be anything, but must be unique with respect to other accounts. The names are purely used for organizational purposes internally.
The PublicKeyPath attribute for a given account specifies the location of the account's public key. Relative paths are resolved relative to the configuration file.
The PrivateKeyPath attribute for a given account specifies the location of the account's private key. Relative paths are resolved relative to the configuration file.
The AcmeURI attribute for a given account specifies the base URI that will be used for ACME operations.

3.5.6. ACME URIs

URI Description
https://acme-staging-v02.api.letsencrypt.org/directory Let's Encrypt staging server
https://acme-v02.api.letsencrypt.org/directory Let's Encrypt production server

3.5.7. Example Accounts

<Accounts>
  <Account Name="main"
           PublicKeyPath="example.pub"
           PrivateKeyPath="example.pri"
           AcmeURI="https://acme-staging-v02.api.letsencrypt.org/directory"/>
</Accounts>
The Outputs element specifies a series of outputs to which issued certificates will be written. Each output has a name, a type, and a set of parameters. Names must be unique with respect to other outputs, and the type must be one of the supported types.

3.6.2. Example Outputs

<Outputs>
  <Output Type="directory"
          Name="main-output">
    <Parameters>
      <Parameter Name="path" Value="/tmp"/>
    </Parameters>
  </Output>
</Outputs>
The DNSConfigurators element specifies a series of external DNS systems in which the certusine client will create records in order to satisfy the DNS challenges posed by the ACME server. Each configurator has a name, a type, and a set of parameters. Names must be unique with respect to other configurators, and the type must be one of the supported types.

3.7.2. Example DNSConfigurators

<DNSConfigurators>
  <DNSConfigurator Type="vultr" Name="vultr-dns">
    <Parameters>
      <Parameter Name="api-key" Value="25DDk6MT+2JI5KBABMysYLPFEOge+MZE3/GiBgrR+CU="/>
      <Parameter Name="domain" Value="example.com"/>
    </Parameters>
  </DNSConfigurator>
</DNSConfigurators>
The Domains element specifies the certificates within domains that the certusine client will attempt to issue and/or renew. A domain has exactly one account, a set of certificate outputs, and exactly one dns configurator. A domain can have any number of certificates.
The Name attribute for a given certificate specifies the name of that certificate. Names can be anything, but must be unique with respect to other certificates. The names are used for organizational purposes internally, and may be used as file names in outputs.
The PublicKeyPath attribute for a given certificate specifies the location of the certificate's public key. Relative paths are resolved relative to the configuration file.
The PrivateKeyPath attribute for a given certificate specifies the location of the certificate's private key. Relative paths are resolved relative to the configuration file.
The Hosts element for a given certificate specifies the hosts within the domain to which the certificate applies. Note that these are not fully-qualified names. Hosts may include wildcards, so for example a certificate may declare a host such as *. To issue a certificate without an explicit hostname (so, for example, to issue a certificate for example.com), simply specify an empty hostname.
The OutputReferences element for a domain specifies the names of the outputs to which certificates will be written. It is an error to specify the name of an undeclared output.
The DNSConfigurator element for a domain specifies the name of the DNS configurator that will be used to complete DNS challenges by the ACME server. It is an error to specify the name of an undeclared DNS configurator.

3.8.8. Example Domains

<Domain Name="example.com"
        Account="main"
        DNSConfigurator="vultr-dns">
  <Certificates>
    <Certificate Name="www"
                 PublicKeyPath="fake.pub"
                 PrivateKeyPath="fake.pri">
      <Hosts>
        <Host Name="www0"/>
        <Host Name="www1"/>
        <Host Name="www2"/>
      </Hosts>
    </Certificate>
    <Certificate Name="mail"
                 PublicKeyPath="fake.pub"
                 PrivateKeyPath="fake.pri">
      <Hosts>
        <Host Name="mail0"/>
        <Host Name="mail1"/>
      </Hosts>
    </Certificate>
    <Certificate Name="wildcard"
                 PublicKeyPath="fake.pub"
                 PrivateKeyPath="fake.pri">
      <Hosts>
        <Host Name="*"/>
      </Hosts>
    </Certificate>
  </Certificates>
  <OutputReferences>
    <OutputReference Name="main-output"/>
  </OutputReferences>
</Domain>
The OpenTelemetry section of the configuration file configures Open Telemetry. This section is optional and telemetry is disabled if the section is not present.
The logical service name should be provided in the LogicalServiceName attribute.
If the OpenTelemetry element contains a Traces element, OTLP traces will be sent to a specified endpoint. The Endpoint attribute specifies the endpoint, and the Protocol attribute can either be GRPC or HTTP.
If the OpenTelemetry element contains a Metrics element, OTLP metrics will be sent to a specified endpoint. The Endpoint attribute specifies the endpoint, and the Protocol attribute can either be GRPC or HTTP.
If the OpenTelemetry element contains a Logs element, OTLP logs will be sent to a specified endpoint. The Endpoint attribute specifies the endpoint, and the Protocol attribute can either be GRPC or HTTP.
An example Open Telemetry configuration:

3.9.5.2. Example

<OpenTelemetry LogicalServiceName="certusine">
  <Logs Endpoint="http://logs.example.com:4317"
        Protocol="GRPC"/>
  <Metrics Endpoint="http://metrics.example.com:4317"
           Protocol="GRPC"/>
  <Traces Endpoint="http://traces.example.com:4317"
          Protocol="GRPC"/>
</OpenTelemetry>
A full configuration file example is as follows:

3.10.2. Example

<?xml version="1.0" encoding="UTF-8" ?>

<Configuration xmlns="urn:com.io7m.certusine:configuration:1">

  <Options CertificateStore="store.db"
           DNSWaitTime="PT5M"
           CertificateExpirationThreshold="PT72H"/>

  <Accounts>
    <Account Name="main"
             PublicKeyPath="fake.pub"
             PrivateKeyPath="fake.pri"
             AcmeURI="https://acme-staging-v02.api.letsencrypt.org/directory"/>
  </Accounts>

  <Outputs>
    <Output Type="directory"
            Name="main-output">
      <Parameters>
        <Parameter Name="path"
                   Value="/tmp"/>
      </Parameters>
    </Output>
  </Outputs>

  <DNSConfigurators>
    <DNSConfigurator Type="vultr"
                     Name="vultr-dns">
      <Parameters>
        <Parameter Name="api-key"
                   Value="NOTANAPIKEY"/>
        <Parameter Name="domain"
                   Value="example.com"/>
      </Parameters>
    </DNSConfigurator>
  </DNSConfigurators>

  <Domains>
    <Domain Name="example.com"
            Account="main"
            DNSConfigurator="vultr-dns">
      <Certificates>
        <Certificate Name="www"
                     PublicKeyPath="fake.pub"
                     PrivateKeyPath="fake.pri">
          <Hosts>
            <Host Name="www0"/>
            <Host Name="www1"/>
            <Host Name="www2"/>
          </Hosts>
        </Certificate>
        <Certificate Name="mail"
                     PublicKeyPath="fake.pub"
                     PrivateKeyPath="fake.pri">
          <Hosts>
            <Host Name="mail0"/>
            <Host Name="mail1"/>
          </Hosts>
        </Certificate>
        <Certificate Name="wildcard"
                     PublicKeyPath="fake.pub"
                     PrivateKeyPath="fake.pri">
          <Hosts>
            <Host Name="*"/>
          </Hosts>
        </Certificate>
      </Certificates>
      <OutputReferences>
        <OutputReference Name="main-output"/>
      </OutputReferences>
    </Domain>
  </Domains>

</Configuration>
The XML schema that defines the configuration file format is as follows:

3.11.2. Schema

<?xml version="1.0" encoding="UTF-8" ?>

<schema xmlns="http://www.w3.org/2001/XMLSchema"
        xmlns:c="urn:com.io7m.certusine:configuration:1"
        targetNamespace="urn:com.io7m.certusine:configuration:1">

  <element name="Options">
    <complexType>
      <attribute name="DNSWaitTime"
                 type="duration"
                 use="optional">
        <annotation>
          <documentation>
            Specifies the amount of time that the client will wait between creating DNS records, and then notifying the
            ACME servers that the records have been created. This wait time is necessary because DNS records sometimes
            take time to propagate, and if the client instructs the ACME server to check the records before they have
            had time to propagate, then the certificate authorization check will fail.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="CertificateStore"
                 type="string"
                 use="required">
        <annotation>
          <documentation>
            Specifies the file that the client will use for its internal database of certificates. Relative paths are
            resolved relative to the configuration file.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="CertificateExpirationThreshold"
                 type="duration"
                 use="optional">
        <annotation>
          <documentation>
            Specifies the maximum amount of time before expiration that the client will allow before it attempts to
            renew a certificate. For example, a value of PT72H means that the client will start attempting to renew a
            certificate when the certificate becomes due to expire in less than 72 hours.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="FaultInjection">
    <annotation>
      <documentation>
        Configuration for fault injection. The purpose of fault injection is to make the program fail in various ways at
        runtime in order to verify that metrics and alerting are working correctly.
      </documentation>
    </annotation>

    <complexType>
      <attribute name="FailTasks"
                 use="optional"
                 type="boolean">
        <annotation>
          <documentation>
            Specifies that failing tasks should be injected into the execution of tasks for each domain renewal.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="FailDNSChallenge"
                 use="optional"
                 type="boolean">
        <annotation>
          <documentation>
            Specifies that DNS challenges should be purposefully failed.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="FailSigningCertificates"
                 use="optional"
                 type="boolean">
        <annotation>
          <documentation>
            Specifies that certificate signing should be purposefully failed.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="CrashTasks"
                 use="optional"
                 type="boolean">
        <annotation>
          <documentation>
            Specifies that crashing tasks should be injected into the execution of tasks for each domain renewal.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="CrashDNSChallenge"
                 use="optional"
                 type="boolean">
        <annotation>
          <documentation>
            Specifies that DNS challenges should be purposefully crashed.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="CrashSigningCertificates"
                 use="optional"
                 type="boolean">
        <annotation>
          <documentation>
            Specifies that certificate signing should be purposefully crashed.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="Account">
    <complexType>
      <attribute name="Name"
                 type="string"
                 use="required">
        <annotation>
          <documentation>
            Specifies the name of that account. Names can be anything, but must be unique with respect to other
            accounts. The names are purely used for organizational purposes internally.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="PublicKeyPath"
                 type="string"
                 use="required">
        <annotation>
          <documentation>
            Specifies the location of the account's public key. Relative paths are resolved relative to the
            configuration file.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="PrivateKeyPath"
                 type="string"
                 use="required">
        <annotation>
          <documentation>
            Specifies the location of the account's private key. Relative paths are resolved relative to the
            configuration file.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="AcmeURI"
                 type="anyURI"
                 use="required">
        <annotation>
          <documentation>
            Specifies the base URI that will be used for ACME operations.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="Accounts">
    <complexType>
      <sequence minOccurs="0"
                maxOccurs="unbounded">
        <element ref="c:Account"/>
      </sequence>
    </complexType>

    <key name="AccountKey">
      <selector xpath="c:Account"/>
      <field xpath="@Name"/>
    </key>
  </element>

  <element name="Parameter">
    <complexType>
      <attribute name="Name"
                 use="required"
                 type="string"/>
      <attribute name="Value"
                 use="required"
                 type="string"/>
    </complexType>
  </element>

  <element name="Parameters">
    <complexType>
      <sequence minOccurs="0"
                maxOccurs="unbounded">
        <element ref="c:Parameter"/>
      </sequence>
    </complexType>

    <key name="ParameterKey">
      <selector xpath="c:Parameter"/>
      <field xpath="@Name"/>
    </key>
  </element>

  <element name="Output">
    <annotation>
      <documentation>
        An output destination for signed certificates.
      </documentation>
    </annotation>

    <complexType>
      <sequence minOccurs="1"
                maxOccurs="1">
        <element ref="c:Parameters"/>
      </sequence>
      <attribute name="Type"
                 type="string"
                 use="required">
        <annotation>
          <documentation>
            The type of the output. Must be one of the supported output types.
          </documentation>
        </annotation>
      </attribute>
      <attribute name="Name"
                 type="string"
                 use="required">
        <annotation>
          <documentation>
            The (unique) name of the output.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="Outputs">
    <annotation>
      <documentation>
        A set of output destinations for signed certificates.
      </documentation>
    </annotation>

    <complexType>
      <sequence minOccurs="0"
                maxOccurs="unbounded">
        <element ref="c:Output"/>
      </sequence>
    </complexType>

    <key name="OutputKey">
      <selector xpath="c:Output"/>
      <field xpath="@Name"/>
    </key>
  </element>

  <element name="DNSConfigurator">
    <annotation>
      <documentation>
        A DNS configurator used to create and delete DNS records in response to ACME challenges.
      </documentation>
    </annotation>

    <complexType>
      <sequence minOccurs="1"
                maxOccurs="1">
        <element ref="c:Parameters"/>
      </sequence>
      <attribute name="Type"
                 type="string"
                 use="required"/>
      <attribute name="Name"
                 type="string"
                 use="required"/>
    </complexType>
  </element>

  <element name="DNSConfigurators">
    <annotation>
      <documentation>
        A set of DNS configurators.
      </documentation>
    </annotation>

    <complexType>
      <sequence minOccurs="0"
                maxOccurs="unbounded">
        <element ref="c:DNSConfigurator"/>
      </sequence>
    </complexType>

    <key name="DNSConfiguratorKey">
      <selector xpath="c:DNSConfigurator"/>
      <field xpath="@Name"/>
    </key>
  </element>

  <element name="Host">
    <annotation>
      <documentation>
        A hostname. Note that this is not a fully-qualified domain name.
      </documentation>
    </annotation>

    <complexType>
      <attribute name="Name"
                 use="required"
                 type="string"/>
    </complexType>
  </element>

  <element name="Hosts">
    <annotation>
      <documentation>
        A set of hostnames.
      </documentation>
    </annotation>

    <complexType>
      <sequence minOccurs="0"
                maxOccurs="unbounded">
        <element ref="c:Host"/>
      </sequence>
    </complexType>

    <unique name="HostsUnique">
      <selector xpath="c:Host"/>
      <field xpath="@Name"/>
    </unique>
  </element>

  <element name="Certificate">
    <annotation>
      <documentation>
        A set of definitions that describe a certificate that will be created and signed.
      </documentation>
    </annotation>

    <complexType>
      <sequence>
        <element ref="c:Hosts"/>
      </sequence>

      <attribute name="Name"
                 use="required"
                 type="string">
        <annotation>
          <documentation>
            The certificate name.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="PublicKeyPath"
                 type="string"
                 use="required">
        <annotation>
          <documentation>
            Specifies the location of the certificate's public key. Relative paths are resolved relative to the
            configuration file.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="PrivateKeyPath"
                 type="string"
                 use="required">
        <annotation>
          <documentation>
            Specifies the location of the certificate's private key. Relative paths are resolved relative to the
            configuration file.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="Certificates">
    <annotation>
      <documentation>
        A set of certificate definitions.
      </documentation>
    </annotation>

    <complexType>
      <sequence minOccurs="0"
                maxOccurs="unbounded">
        <element ref="c:Certificate"/>
      </sequence>
    </complexType>

    <key name="CertificateKey">
      <selector xpath="c:Certificate"/>
      <field xpath="@Name"/>
    </key>
  </element>

  <element name="OutputReference">
    <annotation>
      <documentation>
        A reference to an output definition.
      </documentation>
    </annotation>

    <complexType>
      <attribute name="Name"
                 use="required"
                 type="string"/>
    </complexType>
  </element>

  <element name="OutputReferences">
    <annotation>
      <documentation>
        A set of references to output definitions.
      </documentation>
    </annotation>

    <complexType>
      <sequence minOccurs="1"
                maxOccurs="unbounded">
        <element ref="c:OutputReference"/>
      </sequence>
    </complexType>

    <unique name="OutputReferencesUnique">
      <selector xpath="c:OutputReference"/>
      <field xpath="@Name"/>
    </unique>
  </element>

  <element name="Domain">
    <annotation>
      <documentation>
        A set of definitions that define a domain.
      </documentation>
    </annotation>

    <complexType>
      <sequence>
        <element ref="c:Certificates"/>
        <element ref="c:OutputReferences"/>
      </sequence>

      <attribute name="Name"
                 use="required"
                 type="string">
        <annotation>
          <documentation>
            The fully qualified domain name.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="Account"
                 use="required"
                 type="string">
        <annotation>
          <documentation>
            The account to which this domain belongs.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="DNSConfigurator"
                 use="required"
                 type="string">
        <annotation>
          <documentation>
            The DNS configurator that will be used for this domain.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="Domains">
    <annotation>
      <documentation>
        A set of domain definitions.
      </documentation>
    </annotation>

    <complexType>
      <sequence minOccurs="0"
                maxOccurs="unbounded">
        <element ref="c:Domain"/>
      </sequence>
    </complexType>

    <key name="DomainKey">
      <selector xpath="c:Domain"/>
      <field xpath="@Name"/>
    </key>
  </element>

  <simpleType name="OpenTelemetryProtocol">
    <annotation>
      <documentation>
        The protocol used to deliver OpenTelemetry data.
      </documentation>
    </annotation>

    <restriction base="string">
      <enumeration value="GRPC">
        <annotation>
          <documentation>
            The data will be sent using gRPC.
          </documentation>
        </annotation>
      </enumeration>
      <enumeration value="HTTP">
        <annotation>
          <documentation>
            The data will be sent using HTTP(s).
          </documentation>
        </annotation>
      </enumeration>
    </restriction>
  </simpleType>

  <element name="Metrics">
    <annotation>
      <documentation>
        Configuration information for OpenTelemetry metrics.
      </documentation>
    </annotation>

    <complexType>
      <attribute name="Endpoint"
                 use="required"
                 type="anyURI">
        <annotation>
          <documentation>
            The endpoint to which OTLP metrics data will be sent.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="Protocol"
                 use="required"
                 type="c:OpenTelemetryProtocol">
        <annotation>
          <documentation>
            The protocol used to send metrics data.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="Traces">
    <annotation>
      <documentation>
        Configuration information for OpenTelemetry traces.
      </documentation>
    </annotation>

    <complexType>
      <attribute name="Endpoint"
                 use="required"
                 type="anyURI">
        <annotation>
          <documentation>
            The endpoint to which OTLP trace data will be sent.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="Protocol"
                 use="required"
                 type="c:OpenTelemetryProtocol">
        <annotation>
          <documentation>
            The protocol used to send trace data.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="Logs">
    <annotation>
      <documentation>
        Configuration information for OpenTelemetry logs/events.
      </documentation>
    </annotation>

    <complexType>
      <attribute name="Endpoint"
                 use="required"
                 type="anyURI">
        <annotation>
          <documentation>
            The endpoint to which OTLP log data will be sent.
          </documentation>
        </annotation>
      </attribute>

      <attribute name="Protocol"
                 use="required"
                 type="c:OpenTelemetryProtocol">
        <annotation>
          <documentation>
            The protocol used to send log data.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="OpenTelemetry">
    <annotation>
      <documentation>
        Configuration information for OpenTelemetry.
      </documentation>
    </annotation>

    <complexType>
      <sequence>
        <element ref="c:Logs"
                 minOccurs="0"
                 maxOccurs="1"/>
        <element ref="c:Metrics"
                 minOccurs="0"
                 maxOccurs="1"/>
        <element ref="c:Traces"
                 minOccurs="0"
                 maxOccurs="1"/>
      </sequence>

      <attribute name="LogicalServiceName"
                 use="required"
                 type="string">
        <annotation>
          <documentation>
            The logical name of the service as it will appear in OpenTelemetry.
          </documentation>
        </annotation>
      </attribute>
    </complexType>
  </element>

  <element name="Configuration">
    <annotation>
      <documentation>
        The configuration information for the certusine client.
      </documentation>
    </annotation>

    <complexType>
      <sequence>
        <element ref="c:Options"/>
        <element ref="c:Accounts"/>
        <element ref="c:Outputs"/>
        <element ref="c:DNSConfigurators"/>
        <element ref="c:Domains"/>
        <element ref="c:OpenTelemetry"
                 minOccurs="0"
                 maxOccurs="1"/>
        <element ref="c:FaultInjection"
                 minOccurs="0"
                 maxOccurs="1"/>
      </sequence>
    </complexType>

    <keyref name="DomainAccountsExist"
            refer="c:AccountKey">
      <selector xpath="c:Domains/c:Domain"/>
      <field xpath="@Account"/>
    </keyref>

    <keyref name="DomainDNSConfiguratorsExist"
            refer="c:DNSConfiguratorKey">
      <selector xpath="c:Domains/c:Domain"/>
      <field xpath="@DNSConfigurator"/>
    </keyref>

    <keyref name="OutputReferencesExist"
            refer="c:OutputKey">
      <selector xpath="c:Domains/c:Domain/c:OutputReferences/c:OutputReference"/>
      <field xpath="@Name"/>
    </keyref>
  </element>

</schema>
The certusine package is extensively instrumented with OpenTelemetry in order to allow for the certificate renewal process to be continually monitored. The package publishes metrics, logs, and traces, all of which can be independently enabled or disabled. Most installations will only want to enable metrics or logs in production; traces are more useful when trying to diagnose performance problems, or for doing actual development on the certusine package.
The package publishes the following metrics that can be used for monitoring:

4.2.1.2. Metrics

Name Description
certusine_up A gauge that produces a constant value of 1 for as long as the certusine client is running.
certusine_dns_challenge_failures A counter that is incremented every time a DNS challenge fails during certificate renewal.
certusine_signing_failures A counter that is incremented every time certificate signing fails during certificate renewal.
certusine_renewal_failures A counter that is incremented every time certificate renewal completely fails. This is an important metric for error monitoring; any non-zero value for this metric should be considered a problem.
certusine_renewal_successes A counter that is incremented every time certificate renewal completely succeeds. This is an important metric for error monitoring; if this metric remains continually at zero, this should likely be considered a problem.
certusine_certificates_stored A counter that is incremented every time certificates are stored. The package redundantly stores certificates many times, so this metric will be incremented frequently.
certusine_certificates_store_failures A counter that is incremented every time certificates fail to be stored. If this metric is trending upwards, it may mean that certificates are not being distributed to systems that depend upon them.
certusine_certificate_time_remaining A gauge that displays the age of all the certificates that are currently in the certificate store. Certificates can be distinguished by their domain and certificate name labels/attributes. This is an important metric for alerting.
certusine_certificate_expiration_threshold A gauge that displays the configured expiration threshold in seconds. This is useful when combined with the certusine_certificate_time_remaining metric for alerting.
The package may produce other metrics, however these are undocumented and should not be relied upon.
On a system that supports the querying of metrics using PromQL, alerting rules should be used to notify administrators that certificate renewals are failing.
Assuming that certificate renewal attempts are made at 30 minute intervals, the following rule can be used to capture failures:

4.2.2.1.3. Alert

rate(certusine_renewal_failures_total[30m]) > bool 0
The rate of the certusine_renewal_failures_total metric should be zero over any given interval in a stable system.
Another useful metric to monitor for alerting is the certusine_certificate_time_remaining metric. ACME-issued certificates generally have relatively short expiration times. The de-facto standard Let's Encrypt service issues certificates with a 90-day expiration. If the certusine package is configured to renew any certificate with less than N seconds remaining before expiration (the conventional value being 72 hours, or 259200 seconds), and the expiration time of any certificate drops below, say, N / 2, then this clearly indicates that something is wrong and certificates are not being renewed as expected. An alerting rule for this scenario could be written such as:

4.2.2.1.6. Alert

certusine_certificate_time_remaining{} < 252000
A value of 252000 (70 hours in seconds) is chosen arbitrarily. A working system should have renewed certificates minutes after the 72-hour renewal threshold. This rule could be improved by using the certusine_certificate_expiration_threshold metric:

4.2.2.1.8. Alert

max(certusine_certificate_time_remaining{}) < bool max(certusine_certificate_expiration_threshold{})
The alerting rule would ideally be configured to fire if the above condition remains true for an hour or more. The rule is robust in the face of configuration changes; if the expiration threshold is changed, the value of the certusine_certificate_expiration_threshold metric will automatically update to reflect the change.
The certusine package is conservative in the amount of logging output it produces by default. The package is written to publish only a specific set of log messages to telemetry logs in order to increase the signal-to-noise ratio. The certusine package publishes strongly-typed events internally that are mapped directly to OpenTelemetry log messages, using attributes to hold the fields of the events, and using the certusine.type attribute to hold the name of the event type. The intention is to make it very easy to write alerting rules simply by counting occurrences of log messages that have a given type.
The CSEventCertificateDNSChallengeFailed event type indicates that a DNS challenge failed during certificate renewal.

4.3.2.2. Attributes

Name Description
certusine.type CSEventCertificateDNSChallengeFailed
certusine.domain The domain containing the certificate that failed.
certusine.certificate The name of the certificate that failed.
The CSEventCertificateRenewalFailed event type indicates that a certificate renewal completely failed. All occurrences of this log event should be considered a problem that needs to be fixed.

4.3.3.2. Attributes

Name Description
certusine.type CSEventCertificateRenewalFailed
certusine.domain The domain containing the certificate that failed.
certusine.certificate The name of the certificate that failed.
The CSEventCertificateRenewalSucceeded event type indicates that a certificate renewal completely succeeded.

4.3.4.2. Attributes

Name Description
certusine.type CSEventCertificateRenewalSucceeded
certusine.domain The domain that succeeded.
certusine.certificate The name of the certificate that was renewed.
The CSEventCertificateSigningFailed event type indicates that a certificate could not be signed.

4.3.5.2. Attributes

Name Description
certusine.type CSEventCertificateSigningFailed
certusine.domain The domain containing the certificate that failed.
certusine.certificate The name of the certificate that failed.
The CSEventCertificateStored event type indicates that a certificate was stored.

4.3.6.2. Attributes

Name Description
certusine.type CSEventCertificateStored
certusine.domain The domain containing the certificate that was stored.
certusine.certificate The name of the certificate that was stored.
certusine.target The name of the output to which the certificate was stored.
The CSEventCertificateStoreFailed event type indicates that a certificate could not be stored to an output. Occurrences of this event type may indicate that certificates are not reaching systems that depend on them (for example, if an application is repeatedly failing to publish certificates to directory, applications reading certificates from that directly may start serving expired certificates as newly renewed ones aren't getting through).

4.3.7.2. Attributes

Name Description
certusine.type CSEventCertificateStoreFailed
certusine.domain The domain containing the certificate that could not be stored.
certusine.certificate The name of the certificate that could not be stored.
certusine.target The name of the output to which the certificate could not be stored.
Assuming that certificate renewal attempts are made at 30 minute intervals, the following rule should be used to capture log events of type CSEventCertificateRenewalFailed:

4.3.8.1.2. Alert

count(rate({level="ERROR"} | json | attributes_certusine_type=`CSEventCertificateRenewalFailed`[30m])) > bool 0
The rule captures messages at ERROR severity, unpacks the included attributes using the json operator, and then filters messages based on the generated attributes_certusine_type label. Occurences of matching messages are counted over the previous 30-minute interval, and transformed to a "boolean" numeric value using the greater-than operator.
The certusine package publishes traces for all internal operations. No specific documentation is provided on the structures of the traces as they are effectively tied to the internal structure of the code and are subject to change.
This section of the documentation describes all of the supported certificate output types.
Directory - Write certificates to a local directory
The Directory output writes certificates to a local directory. For a directory at /path, for a given domain name d and certificate name c, certificates are written to the following files:

5.2.2.2. Output Keys

Key Name Description
/path/d/c/public.key The certificate public key.
/path/d/c/private.key The certificate private key.
/path/d/c/certificate.pem The PEM-encoded certificate.
/path/d/c/full_chain.pem The PEM-encoded full certificate chain.
Note that d is percent-encoded to ensure that names are filesystem-safe.
The output accepts the following parameters:

5.2.2.5. Parameters

Parameter Type Required Description
path Path true The path to the output directory.

5.2.3.1. Example Configuration

<Output Type="Directory" Name="an-example-output">
  <Parameters>
    <Parameter Name="path" Value="/tmp"/>
  </Parameters>
</Output>
Looseleaf - Write certificates to a looseleaf server
The Looseleaf output writes certificates to an external looseleaf server. For a given domain name d and certificate name c, certificates are written to the following database keys:

5.3.2.2. Output Keys

Key Name Description
/certificates/d/c/public_key The certificate public key.
/certificates/d/c/private_key The certificate private key.
/certificates/d/c/certificate The PEM-encoded certificate.
/certificates/d/c/certificate_full_chain The PEM-encoded full certificate chain.
The output accepts the following parameters:

5.3.2.4. Parameters

Parameter Type Required Description
password Password false The password used to authenticate with the looseleaf server.
username User name false The username used to authenticate with the looseleaf server.
endpoint URI true The base address/URI of the looseleaf server.

5.3.3.1. Example Configuration

<Output Type="Looseleaf" Name="an-example-looseleaf-output">
  <Parameters>
    <Parameter Name="endpoint" Value="http://looseleaf.example.com:20000/"/>
    <Parameter Name="username" Value="grouch"/>
    <Parameter Name="password" Value="12345678"/>
  </Parameters>
</Output>
This section of the documentation describes all the supported DNS configurator types.
Gandi-v5 - Support for the Gandi LiveDNS API
The Gandi-v5 DNS configurator supports creating and deleting DNS records using the Gandi DNS API.
The output accepts the following parameters:

6.2.2.3. Parameters

Parameter Type Required Description
domain Domain name true The domain name.
personal-access-token Personal access token true A generated Gandi personal access token.
api-base URI false The Gandi API base address.
To use the production Gandi API, use the base URI https://api.gandi.net.
Hetzner - Support for the Hetzner DNS API
The Hetzner DNS configurator supports creating and deleting DNS records using the Hetzner DNS API.
The output accepts the following parameters:

6.3.2.3. Parameters

Parameter Type Required Description
zone-id Zone ID true The DNS zone ID.
domain-name Hetzner Zone ID true The Hetzner domain zone ID.
api-key API Key true The Hetzner API key.
api-base URI false The Hetzner API base address.
If not specified, the configurator defaults to the production API base address https://dns.hetzner.com/api/v1/.
Vultr - Support for the Vultr DNS API
The Vultr DNS configurator supports creating and deleting DNS records using the Vultr DNS API.
The output accepts the following parameters:

6.4.2.3. Parameters

Parameter Type Required Description
domain Domain name true The domain name.
api-key API Key true The Vultr API key.
api-base URI false The Vultr API base address.
The certusine package provides a command-line interface for performing tasks such as starting the server, checking configuration files, generating keypairs, etc. The base certusine command is broken into a number of subcommands which are documented over the following sections.

7.1.2. Command-Line Overview

certusine: usage: certusine [command] [arguments ...]

  The certusine ACME client.

  Use the "help" command to examine specific commands:

    $ certusine help help.

  Command-line arguments can be placed one per line into a file, and
  the file can be referenced using the @ symbol:

    $ echo help > file.txt
    $ echo help >> file.txt
    $ certusine @file.txt

  Commands:
    check-configuration         Check configuration file.
    generate-keypair            Generate keypairs.
    help                        Show usage information for a command.
    looseleaf-download          Download certificates from looseleaf databases.
    renew                       Renew certificates.
    show-certificate-outputs    Show supported certificate outputs.
    show-dns-configurators      Show supported DNS configurators.
    version                     Show the application version.

  Documentation:
    https://www.io7m.com/software/certusine/
All subcommands accept a --verbose parameter that may be set to one of trace, debug, info, warn, or error. This parameter sets the lower bound for the severity of messages that will be logged. For example, at debug verbosity, only messages of severity debug and above will be logged. Setting the verbosity to trace level effectively causes everything to be logged, and will produce large volumes of debugging output.
The certusine command-line tool uses quarrel to parse command-line arguments, and therefore supports placing command-line arguments into a file, one argument per line, and then referencing that file with @. For example:

7.1.5. @ Syntax

$ certusine check-configuration --file config.xml

$ (cat <<EOF
check-configuration
--file
config.xml
EOF
) > args.txt

$ certusine @args.txt
All subcommands, unless otherwise specified, yield an exit code of 0 on success, and a non-zero exit code on failure.
check-configuration - Validate configuration files
The check-configuration command validates configuration files.

7.2.2.2. Parameters

Parameter Type Required Description
--verbose CLPLogLevel false Set the minimum logging verbosity level.
--file Path true The configuration file
The check-configuration command will validate the configuration file specified with --file.
If the command encounters no errors or warnings, it will not print anything.

7.2.3.1. Example

$ certusine check-configuration --file main.xml
error: main.xml:23:5: Multiple DNS configurators defined with the name "vultr-dns"
generate-keypair - Generate keypairs
The generate-keypair command generates keypairs.

7.3.2.2. Parameters

Parameter Type Required Description
--verbose CLPLogLevel false Set the minimum logging verbosity level.
--public-key Path true The public key
--private-key Path true The private key
--overwrite boolean false Overwrite keys if already present.
The generate-keypair command will generate a keypair and write the private key to the file specified with --private-key, and write the public key to the file specified with --public-key. If the key files already exist, they will not be overwritten unless the --overwrite parameter is specified.
If the command encounters no errors or warnings, it will not print anything.

7.3.3.1. Example

$ certusine generate-keypair --private-key key.pri --public-key key.pub

$ certusine generate-keypair --private-key key.pri --public-key key.pub
error: java.nio.file.FileAlreadyExistsException: key.pub

$ certusine generate-keypair --private-key key.pri --public-key key.pub --overwrite true
renew - Issue and renew certificates
The renew command encapsulates the main functionality of the certusine package: It issues and/or renews certificates in a perpetual loop, and sends those certificates to a set of configured outputs.

7.4.2.2. Parameters

Parameter Type Required Description
--file Path true The configuration file
--only-once boolean false Renew certificates once and then exit.
--schedule Duration false Renew certificates repeatedly, waiting this duration between attempts.
--verbose CLPLogLevel false Set the minimum logging verbosity level.
The renew command will validate the configuration file specified with --file, and then loop forever attempting to issue and/or renew all certificates for all domains specified in the domains section of the configuration file. The command will pause for the duration specified by --schedule between attempts to renew certificates, and only certificates that have less than the amount of time specified by the CertificateExpirationThreshold parameter before expiration will be passed through the full ACME renewal process. If the --only-once option is specified, the client will execute one iteration of the renewal loop and then exit.
The renew command will aggressively write and re-write existing certificates to all the configured certificate outputs, redundantly. Thus, it is necessary that all certificate output implementations be idempotent with regards to the write operation. There are multiple reasons for this redundancy. Firstly, the certusine client cannot know the status of all the external systems to which it supplies certificates; external systems can be destroyed and recreated at any given time, and an external system should not be forced to wait until the next certificate renewal to receive certificates just because it wasn't present at the exact time the original issue/renewal occurred. Secondly, the very nature of systems being external means that the act of sending certificates to those systems can fail. Whilst the certusine client does retry I/O operations on failure, sometimes a system can be inaccessible long enough for all of the retry attempts to fail. If the certusine client did not redundantly write certificates, the external system would be stuck without any certificates until the next full renewal attempt.

7.4.4.1. Example

$ certusine renew --file config.xml --schedule PT1M
info: [example.com] (attempt 1/10) checking if domain is authorized
info: [example.com] (attempt 1/10) domain is already authorized
info: [example.com] (attempt 1/10) checking if certificates require reissuing
info: [example.com] (attempt 1/10) certificates do not require reissuing
info: [example.com] (attempt 1/10) saving certificates to outputs
info: [example.com] (attempt 1/10) saving certificate to output directory:main-output
info: waiting until 2022-07-01T13:45:34Z for the next renewal attempt (PT1M)
show-certificate-outputs - Show supported certificate outputs
The show-certificate-outputs command displays the supported certificate outputs.

7.5.2.2. Parameters

Parameter Type Required Description
--verbose CLPLogLevel false Set the minimum logging verbosity level.
The show-certificate-outputs command will display a list of supported certificate outputs. The values printed are suitable for use as type parameters in output declarations.

7.5.3.1. Example

$ certusine show-certificate-outputs
Looseleaf : Write certificates to a looseleaf server.
Directory : Write certificates to a local directory.
show-dns-configurators - Show supported DNS configurators
The show-dns-configurators command displays the supported DNS configurators.

7.6.2.2. Parameters

Parameter Type Required Description
--verbose CLPLogLevel false Set the minimum logging verbosity level.
The show-dns-configurators command will display a list of supported DNS configurators. The values printed are suitable for use as type parameters in dns configurator declarations.

7.6.3.1. Example

$ certusine show-dns-configurators
Gandi-v5 : Configure DNS records using the Gandi LiveDNS v5 API.
Vultr : Configure DNS records using the Vultr DNS API.
looseleaf-download - Download certificates from a looseleaf server
The looseleaf-download command downloads certificates from a looseleaf server. The command is designed to run perpetually as a service, repeatedly downloading certificates on a schedule.

7.7.2.2. Parameters

Parameter Type Required Description
--endpoint String true The target looseleaf endpoint base.
--output-directory Path true The output directory.
--domain String true The domain name.
--username String true The user name.
--password String true The password.
--certificate-name List<String> true The certificate name(s). May be specified multiple times.
--only-once boolean false Download certificates once and then exit.
--schedule Duration false Download certificates repeatedly, waiting this duration between attempts.
--verbose CLPLogLevel false Set the minimum logging verbosity level.

7.7.3.1. Example

$ certusine looseleaf-download               \
  --endpoint http://looseleaf4.example.com/  \
  --output-directory /certificates           \
  --domain example.com                       \
  --certificate-name www                     \
  --certificate-name mail                    \
  --username somebody                        \
  --password 12345678                        \
  --schedule PT1H
version - Display the certusine version
The version command displays the current version of the command-line tool.

7.8.2.2. Parameters

Parameter Type Required Description
--verbose CLPLogLevel false Set the minimum logging verbosity level.

7.8.3.1. Example

$ certusine version
com.io7m.certusine 2.0.0-SNAPSHOT b2163bef9f4870ea97a97aad21f76667387a1fc3
io7m | single-page | multi-page | epub | Certusine User Manual