The certusine client is an ACME client with a focus on minimalism, a
small footprint, and reliability. The client has the following notable features:
The certusine package is available from the following sources:
Regardless of the distribution method, the certusine package
will contain a command named certusine that acts as the main
entrypoint to all the package's functionality. The certusine
command expects an environment variable named CERTUSINE_HOME
to be defined that points to the installation directory. See the documentation for
the
installation methods below for details.
The certusine command requires that a Java 21+ compatible
JVM be accessible via /usr/bin/env java.
Verify the integrity of the distribution zip file:
Unzip the zip file, and set CERTUSINE_HOME appropriately:
The
certusine package uses
SQLite internally. The
certusine package is written in
Java and includes native
SQLite libraries compiled for a wide range of platforms. At run-time, the native library
for
the current platform is extracted to a temporary directory. By default, the standard
temporary
directory specified by the
java.io.tmpdir system property is
used (
/tmp on most Unix-like systems).
On systems using
SELinux,
applications are not allowed to execute code from binaries or libraries that are in
a directory
that has the
tmp_t SELinux type. It is practically guaranteed that
any system-wide temporary directory will be configured to have this type.
Therefore, in order to run
certusine on these systems,
it is necessary to specify a directory using the
org.sqlite.tmpdir
system property. As an example, the
OCI image
configures this property to use
/certusine/var to store the temporary
native libraries.
Versions of the
certusine package prior to
3.0.0
used the
MVStore database
to store internal certificate state. In version
3.0.0, the
certusine package switched to using
SQLite.
Please either delete the
existing store before
running
3.0.0 on a system that previously used
2.*.*, or specify a new file name for the store in the
configuration file. Certificates will be automatically recreated and placed into the
new store
on application startup.
The certusine client accepts an XML configuration file with a very strictly-defined
schema. The configuration file is an XML element with the following elements:
The
certusine package exclusively uses
ISO 8601
syntax when specifying durations of time. For example, the string
PT1M specifies a
one-minute duration. The string
PT1H specifies a one-hour duration.
The xmlns attribute must be present and set to the value
"urn:com.io7m.certusine:configuration:1".
The Options element specifies global options for the client.
The DNSWaitTime attribute specifies the amount of time that the client will wait
between creating DNS records, and then notifying the ACME servers that the records
have been created. This wait
time is necessary because DNS records sometimes take time to propagate, and if the
client instructs the ACME
server to check the records before they have had time to propagate, then the certificate
authorization check will
fail.
The CertificateStore attribute specifies the file that the client will use for its
internal database of certificates. Relative paths are resolved relative to the configuration
file.
The CertificateExpirationThreshold attribute specifies the maximum amount of time
before expiration that the client will allow before it attempts to renew a certificate.
For example, a value of
PT72H means that the client will start attempting to renew a certificate when
the certificate becomes due to expire in less than 72 hours.
The Accounts element specifies a set of ACME accounts. Most installations will
only use a single account.
The Name attribute for a given account specifies the name of that account. Names
can be anything, but must be unique with respect to other accounts. The names are
purely used for organizational
purposes internally.
The PublicKeyPath attribute for a given account specifies the location of the
account's public key. Relative paths are resolved relative to the configuration file.
The PrivateKeyPath attribute for a given account specifies the location of the
account's private key. Relative paths are resolved relative to the configuration file.
The AcmeURI attribute for a given account specifies the base URI that will be used
for ACME operations.
The
Outputs element specifies a series of outputs to which issued certificates
will be written. Each output has a name, a type, and a set of parameters. Names must
be unique with respect to
other outputs, and the type must be one of the
supported types.
The
DNSConfigurators element specifies a series of external DNS systems in which
the
certusine client will create records in order to satisfy the DNS challenges posed
by the ACME server. Each configurator has a name, a type, and a set of parameters.
Names must be unique with
respect to other configurators, and the type must be one of the
supported types.
The
Domains element specifies the certificates within domains that the
certusine client will attempt to issue and/or renew. A domain has exactly one
account, a set of
certificate outputs, and
exactly one
dns configurator. A domain
can have any number of
certificates.
The
Name attribute for a given certificate specifies the name of that certificate.
Names can be anything, but must be unique with respect to other certificates. The
names are used for
organizational purposes internally, and may be used as file names in
outputs.
The PublicKeyPath attribute for a given certificate specifies the location of the
certificate's public key. Relative paths are resolved relative to the configuration
file.
The PrivateKeyPath attribute for a given certificate specifies the location of the
certificate's private key. Relative paths are resolved relative to the configuration
file.
The Hosts element for a given certificate specifies the hosts within the domain
to which the certificate applies. Note that these are not fully-qualified names.
Hosts may include wildcards, so for example a certificate may declare a host
such as *. To issue a certificate without an explicit hostname (so, for example,
to issue a certificate for example.com), simply specify an empty hostname.
The
OutputReferences element for a domain specifies the names of the
outputs to which certificates will be written.
It is an error to specify the name of an undeclared output.
The
DNSConfigurator element for a domain specifies the name of the
DNS configurator that will be used to complete
DNS challenges by the ACME server. It is an error to specify the name of an undeclared
DNS configurator.
The
OpenTelemetry section of the configuration file configures
Open Telemetry. This section is optional and
telemetry is disabled if the section is not present.
The logical service name should be provided in the LogicalServiceName
attribute.
If the
OpenTelemetry element contains a
Traces element, OTLP
traces
will be sent to a specified endpoint. The
Endpoint attribute
specifies the endpoint, and the
Protocol attribute can either
be
GRPC or
HTTP.
If the
OpenTelemetry element contains a
Metrics element, OTLP
metrics
will be sent to a specified endpoint. The
Endpoint attribute
specifies the endpoint, and the
Protocol attribute can either
be
GRPC or
HTTP.
If the
OpenTelemetry element contains a
Logs element, OTLP
logs
will be sent to a specified endpoint. The
Endpoint attribute
specifies the endpoint, and the
Protocol attribute can either
be
GRPC or
HTTP.
An example Open Telemetry configuration:
A full configuration file example is as follows:
The
XML schema that defines the configuration file
format is as follows:
The
certusine package is extensively instrumented with
OpenTelemetry
in order to allow for the certificate renewal process to be continually monitored.
The package publishes
metrics,
logs,
and
traces, all of which
can be independently enabled or disabled. Most installations will only want to
enable metrics or logs in production; traces are more useful when trying to diagnose
performance
problems, or for doing actual development on the
certusine package.
The package publishes the following metrics that can be used for monitoring:
The package may produce other metrics, however these are undocumented and
should not be relied upon.
On a system that supports the querying of metrics using
PromQL,
alerting rules should be used to notify administrators that certificate renewals are
failing.
Assuming that certificate renewal attempts are made at 30 minute intervals,
the following rule can be used to capture failures:
The rate of the certusine_renewal_failures_total metric should be
zero over any given interval in a stable system.
Another useful metric to monitor for alerting is the
certusine_certificate_time_remaining metric. ACME-issued certificates
generally have relatively short expiration times. The de-facto standard
Let's Encrypt service issues
certificates with a 90-day expiration. If the
certusine package
is configured to renew any certificate with less than
N seconds
remaining before expiration (the conventional value being 72 hours, or 259200 seconds),
and the
expiration time of any certificate drops below, say,
N / 2, then
this clearly indicates that something is wrong and certificates are not being renewed
as expected.
An alerting rule for this scenario could be written such as:
A value of 252000 (70 hours in seconds) is chosen arbitrarily. A working
system should have renewed certificates minutes after the 72-hour renewal threshold.
This rule could be
improved by using the certusine_certificate_expiration_threshold
metric:
The alerting rule would ideally be configured to fire if the above condition remains
true for an hour or more.
The rule is robust in the face of configuration changes; if the expiration threshold
is changed, the
value of the certusine_certificate_expiration_threshold metric will
automatically update to reflect the change.
The certusine package is conservative in the amount of logging output
it produces by default. The package is written to publish only a specific set of log
messages to
telemetry logs in order to increase the signal-to-noise ratio. The
certusine package publishes strongly-typed events internally that
are mapped directly to OpenTelemetry log messages, using attributes to hold the fields
of the
events, and using the certusine.type attribute to hold the name
of the event type. The intention is to make it very easy to write alerting rules simply
by counting
occurrences of log messages that have a given type.
The CSEventCertificateDNSChallengeFailed event type indicates that a
DNS challenge failed during certificate renewal.
The CSEventCertificateRenewalFailed event type indicates that a
certificate renewal completely failed. All occurrences of this log event should be
considered
a problem that needs to be fixed.
The CSEventCertificateRenewalSucceeded event type indicates that a
certificate renewal completely succeeded.
The CSEventCertificateSigningFailed event type indicates that a
certificate could not be signed.
The CSEventCertificateStored event type indicates that a
certificate was stored.
The CSEventCertificateStoreFailed event type indicates that a
certificate could not be stored to an output. Occurrences of this event type may indicate
that certificates are not reaching systems that depend on them (for example, if an
application
is repeatedly failing to publish certificates to directory, applications reading certificates
from that directly may start serving expired certificates as newly renewed ones aren't
getting
through).
Assuming that certificate renewal attempts are made at 30 minute intervals,
the following rule should be used to capture log events of type
CSEventCertificateRenewalFailed:
The rule captures messages at ERROR severity, unpacks the included
attributes using the json operator, and then filters messages
based on the generated attributes_certusine_type label. Occurences
of matching messages are counted over the previous 30-minute interval, and transformed
to a
"boolean" numeric value using the greater-than operator.
The certusine package publishes traces for all internal operations. No
specific documentation is provided on the structures of the traces as they are effectively
tied to
the internal structure of the code and are subject to change.
This section of the documentation describes all of the supported certificate output
types.
Directory - Write certificates to a local directory
The Directory output writes certificates to a local directory.
For a directory at /path, for a given domain name d
and certificate name c, certificates are written to the following files:
The output accepts the following parameters:
Looseleaf - Write certificates to a looseleaf server
The
Looseleaf output writes certificates to an external
looseleaf
server. For a given domain name
d and certificate name
c, certificates are written to the following database keys:
The output accepts the following parameters:
This section of the documentation describes all the supported DNS configurator types.
Gandi-v5
- Support for the Gandi LiveDNS API
The
Gandi-v5 DNS configurator supports creating and deleting DNS records using
the
Gandi DNS API.
The output accepts the following parameters:
Vultr
- Support for the Vultr DNS API
The
Vultr DNS configurator supports creating and deleting DNS records using
the
Vultr DNS API.
The output accepts the following parameters:
The certusine package provides a command-line interface for performing tasks such as
starting the server, checking configuration files, generating keypairs, etc. The base
certusine command is broken into a number of subcommands which are documented
over the following sections.
All subcommands accept a --verbose parameter that may be set to one of
trace, debug, info,
warn, or error. This parameter sets the lower bound for
the severity of messages that will be logged. For example, at debug verbosity, only
messages of severity debug and above will be logged. Setting the verbosity to
trace
level effectively causes everything to be logged, and will produce large volumes of
debugging output.
The
certusine command-line tool uses
quarrel to parse
command-line arguments, and therefore supports placing command-line arguments into
a file,
one argument per line, and then referencing that file with
@. For example:
All subcommands, unless otherwise specified, yield an exit code of 0 on success, and
a non-zero exit code on failure.
check-configuration - Validate configuration files
The check-configuration command validates configuration files.
The check-configuration command will validate the configuration file specified with
--file.
If the command encounters no errors or warnings, it will not print anything.
generate-keypair - Generate keypairs
The generate-keypair command generates keypairs.
The generate-keypair command will generate a keypair and write the private key
to the file specified with --private-key, and write the public key to the file
specified with --public-key. If the key files already exist, they will not be
overwritten unless the --overwrite parameter is specified.
If the command encounters no errors or warnings, it will not print anything.
renew
- Issue and renew certificates
The renew command encapsulates the main functionality of the
certusine
package: It issues and/or renews certificates in a perpetual loop, and sends those
certificates to a set of
configured outputs.
The
renew command will validate the configuration file specified with
--file, and then loop forever attempting to issue and/or renew all certificates for
all domains specified in the
domains
section of the configuration file. The command will pause for the duration specified
by
--schedule
between attempts to renew certificates, and only certificates that have less than
the amount of time specified by
the
CertificateExpirationThreshold
parameter before expiration will be passed through the full
ACME
renewal process. If the
--only-once option is specified, the client will execute one
iteration of the renewal loop and then exit.
The
renew command will aggressively write and re-write existing certificates to all
the configured certificate
outputs, redundantly. Thus, it is necessary that all
certificate output implementations be
idempotent with regards to the write operation.
There are multiple reasons for this redundancy. Firstly, the
certusine
client cannot know the status of all the external systems to which it supplies certificates;
external systems can
be destroyed and recreated at any given time, and an external system should not be
forced to wait until the next
certificate renewal to receive certificates just because it wasn't present at the
exact time the original
issue/renewal occurred. Secondly, the very nature of systems being external means
that the act of sending
certificates to those systems can fail. Whilst the
certusine client does retry I/O
operations on failure, sometimes a system can be inaccessible long enough for all
of the retry attempts to fail.
If the
certusine
client did not redundantly write certificates, the external system would be stuck
without any certificates until
the next full renewal attempt.
show-certificate-outputs - Show supported certificate outputs
The show-certificate-outputs command displays the supported certificate outputs.
The
show-certificate-outputs command will display a list of supported certificate
outputs. The values printed are suitable for use as
type parameters in
output declarations.
show-dns-configurators - Show supported DNS configurators
The show-dns-configurators command displays the supported DNS configurators.
The
show-dns-configurators command will display a list of supported DNS
configurators. The values printed are suitable for use as
type parameters in
dns configurator declarations.
looseleaf-download - Download certificates from a looseleaf server
The
looseleaf-download command downloads certificates from a
looseleaf server.
The command is designed to run perpetually as a service, repeatedly downloading certificates
on a schedule.
version - Display the certusine version
The version command displays the current version of the command-line tool.