CertPilot Docs

Appearance

Operations ​

Running CertPilot, and what to do when something goes wrong.

First run ​

Four things, in order.

1. Generate a KEK ​

bash
make generate-kek

Store it somewhere durable before issuing anything. Every certificate private key and CA credential is encrypted with it, and there is no recovery path. An environment variable injected from a secret manager is the minimum; a .env file on one laptop is not a plan.

bash
export CERTPILOT_KEK='...'

2. Point at a database ​

bash
export CERTPILOT_DB_URL='postgres://user:pass@host:5432/certpilot?sslmode=require'

Any PostgreSQL 13 or later — the schema uses gen_random_uuid(), which is built in from 13. Tested against 17. You run the server; CertPilot does not provision one — see database.md for the pooler note, which matters.

Without a connection string the core runs on the in-memory store with sample data, which is the right way to spend ten minutes on it and the wrong way to run it.

3. Apply migrations ​

bash
make migrate

4. Generate the gateway channel material ​

bash
make dev-certs

This writes development mTLS material into .certpilot/pki/. In production, issue the core and each gateway a certificate from your own internal CA and point --tls-ca at that CA. The development material is a convenience for getting the shape right, not something to deploy.

Migrations ​

Numbered SQL files in migrations/, applied in order, each recorded in schema_migrations with a checksum.

bash
make migrate                                     # uses CERTPILOT_DB_URL
make migrate DB='postgres://...'                 # explicit
docker compose -f deploy/docker-compose.yml --profile migrate run --rm migrate

Three rules the migrator enforces or expects:

They are append-only. A file that has been applied is never edited. To change something, add a migration.

They are rerunnable. Every statement is IF NOT EXISTS, or guarded on the catalogue. PostgreSQL has no ADD CONSTRAINT IF NOT EXISTS, so constraints are wrapped in a do $$ … pg_constraint … $$ block. Migration 002 was not rerunnable for twenty-three migrations and nothing noticed until a conformance suite applied the whole schema twice.

A changed checksum is reported, not re-run. If a file changes after being applied, the migrator warns and moves on:

WARNING  002_crypto_agility.sql has changed since it was applied to this database.
         It was not rerun. Reconcile the difference with a new migration.

This warning is currently expected once on any database migrated before August 2026: migration 002 was fixed to be rerunnable, which changed its checksum. It is recorded as applied and will not re-run. See database.md.

The server never applies migrations itself. A schema change is something an operator runs, not a side effect of a replica restarting during a deploy.

Deploying ​

Docker Compose ​

bash
cp deploy/compose.env.example deploy/.env               # edit it
cp deploy/config.production.example.yaml deploy/config.production.yaml
make dev-certs                                          # or your own CA material
docker compose -f deploy/docker-compose.yml --env-file deploy/.env up -d

Four containers: core, ACME gateway, Vault gateway, frontend. The database is deliberately not in the compose file — CertPilot stores private keys, and running its store as an ephemeral container beside the app is how an evaluation becomes a deployment nobody meant to make.

The compose stack and Dockerfiles were repaired in this pass — they pinned Go 1.24 against a workspace requiring 1.26.6, so no image had built since the version bump, and the core had no CERTPILOT_KEK so it could not have started against a database. They have not been verified by an actual build, because no container runtime was available on the machine where they were fixed. Treat the first docker compose build as the test.

Systemd ​

The core is a single static binary with no runtime dependencies beyond CA certificates.

ini
[Unit]
Description=CertPilot Core
After=network-online.target

[Service]
Type=simple
User=certpilot
Environment=CERTPILOT_KEK=...
Environment=CERTPILOT_DB_URL=...
ExecStart=/usr/local/bin/certpilot-core --config=/etc/certpilot/config.yaml
Restart=always
RestartSec=5

# The core needs to read its config and mTLS material and nothing else.
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadOnlyPaths=/etc/certpilot

[Install]
WantedBy=multi-user.target

Prefer EnvironmentFile= with 0600 permissions over inline Environment= for the KEK — the latter is visible in systemctl show.

Each gateway is the same shape on its own port.

Behind a reverse proxy ​

One thing matters and it is easy to get wrong: the SSE endpoint must not be buffered.

nginx
location /api/v1/events {
    proxy_pass http://core:8080/api/v1/events;
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 24h;
    chunked_transfer_encoding off;
}

Without proxy_buffering off, nginx holds the stream until a buffer fills. A dashboard that has stopped updating then looks merely quiet — which on a wall display is the worst available failure, because it reports "no problems" precisely when the tool has stopped being able to see problems. The shipped deploy/docker/nginx.conf has this block, and it must come before the generic /api/ one.

The core also sends X-Accel-Buffering: no on that route as a second line of defence.

Running more than one replica ​

Run as many as you like. There is no leader election and none is needed.

  • Enqueues collide on a partial unique index with ON CONFLICT DO NOTHING, so two schedulers noticing the same due certificate produce one job.
  • Claims use SELECT … FOR UPDATE SKIP LOCKED, so two workers claiming produce one execution.
  • A worker that dies mid-job holds a lease that expires; another picks it up.

This is deliberate. A leader has a failover window during which nothing renews, and for a system whose purpose is that certificates do not expire, that is the wrong failure mode.

What is not shared between replicas: the SSE event broker is in-process, so a browser sees events from the replica it is connected to. Events are published on the replica doing the work, so an alert is raised once, by whichever replica ran the job — but a dashboard pinned to a different replica will not see that event until it refreshes its snapshot. Sticky sessions on /api/v1/events are worth configuring if you run several.

Rotating the KEK ​

Rotation changes which key seals new data, without downtime, because each envelope and each audit entry records the key that wrote it. It does not let you retire the old key: keep that for as long as you keep the audit log.

1. Make the new key primary and keep the old one as retired.

bash
export CERTPILOT_KEK='<new key>'
export CERTPILOT_KEK_RETIRED='<old key>'

Restart the core. From this point new writes are sealed under the new key, and everything sealed under the old one still reads.

2. Know what moves to the new key, and when. A record is re-sealed only when it is written again:

RecordRe-sealed under the new key
Private keys of certificates CertPilot holdsWhen the certificate renews, which generates a new key
Deployment targets, notification channels, cloud connectionsWhen updated with config sent again. An update without it keeps the stored ciphertext as it is
CA accountsNever. There is no update route, so the only way is to recreate the account
Audit entriesNever. Each is signed once, by the key in use when it was written

Nothing reports which records still depend on the retired key, and there is no bulk re-seal command.

3. Keep the retired key. Checking the audit chain needs every key that ever signed an entry. Drop one, and /api/v1/audit/verify reports the chain unverifiable from the first entry that key signed; anything still sealed under it becomes unreadable too. Measured on v0.2.1: after a rotation, dropping the retired key broke the chain at entry 1, private-key export answered 500, and issuing through a CA account created before the rotation failed to decrypt its configuration.

CERTPILOT_KEK_RETIRED accepts a comma-separated list, so more than one generation can be kept.

After a suspected compromise of the old key, rotation protects only what is written from then on. Everything still sealed under the old key, and every audit entry it signed, is as exposed as the key is: reissue the certificates CertPilot holds keys for, and replace the credentials in CA accounts, deployment targets, notification channels and cloud connections at their source.

Backups and restore ​

Back up the database. That is the whole of the state.

bash
pg_dump "$CERTPILOT_DB_URL" --format=custom --file=certpilot-$(date +%F).dump

A backup without the KEK is not a backup. The dump contains ciphertext for every private key and CA credential. Store the KEK separately — a backup and the key to it in the same place is one compromise, not two.

Restore:

bash
pg_restore --dbname="$CERTPILOT_DB_URL" --clean --if-exists certpilot-2026-08-22.dump
export CERTPILOT_KEK='<the key that sealed it>'

Then check the estate reconciles: the CA health sweep and the renewal scheduler both run on startup, so a restored database converges without intervention. Restored with the key that sealed it, the database reads as it did: the audit chain checks, the private keys CertPilot holds still match their certificates, and renewal and issuance work.

Given the wrong key, the core refuses to start. It compares the key that signed the newest audit entry with the keys it was given, and stops before writing anything if none of them matches:

this database was sealed with key encryption key cb9cffa7e89e96e5, which signed
its newest audit entry (4), and this core was given key b09f5d25a3121a51. …

Core v0.2.1 and earlier started anyway, and could read nothing they had sealed. The first sign-in then wrote an audit entry signed with the wrong key, which could never be checked once the right key was back, so the audit chain reported itself broken from then on.

If the key is lost for good, everything it sealed is lost with it: the private keys of certificates CertPilot holds, and the configuration of every CA account, deployment target, notification channel and cloud connection. Restart once with the lost key's identifier, as printed in the refusal:

bash
export CERTPILOT_KEK='<a new key>'
export CERTPILOT_KEK_ABANDON=cb9cffa7e89e96e5

The core starts, and writes a secrets.kek_abandoned audit entry, signed with the new key, naming the one given up. Remove the variable afterwards: the next start no longer needs it. Then recreate those accounts, targets, channels and connections, and reissue the certificates whose keys CertPilot held. Certificates held by agents are unaffected, because their keys never left the host. The audit chain stays readable, but entries signed with the lost key can no longer be checked, so /api/v1/audit/verify reports the chain from its first entry as unverifiable.

What to keep, and what can be reissued ​

StateWhere it livesIf it is lost
Certificates, CA accounts, templates, agents, the audit logThe databaseRestore the backup
The KEKYour secret manager, never beside the backupEverything it sealed is lost; see above
ACME account keysaccount_key_pem in the CA account's configuration, sealed in the database, if you set one; otherwise the ACME gateway's --state-dirA new account is registered with the CA. One that requires External Account Binding needs EAB credentials again
Agent identitiesEach agent's --state-dir on its own hostRe-enrol that host
The self-signed gateway's CANowhere: it is in memory by design, and a restart issues from a new authorityNothing to keep
Vault credentialsThe CA account's configuration, sealed in the databaseRestore the backup
Gateway mTLS materialYour internal CAReissue it

Upgrading ​

bash
git pull
make build
make migrate          # if the release adds migrations
# restart the core, then the gateways

Order matters in one direction only: migrate before starting the new core. A core started on an older schema refuses, naming both versions:

this database's schema is at migration 039, and this core needs migration 043.
Run the migrations before starting it: certpilot-core --migrate, …

Core v0.2.1 and earlier started anyway, reported themselves healthy, and renewed nothing until somebody read the log. A newer schema than the core needs is accepted, so rolling the binary back after a migration still starts. Gateways are independent and can be restarted whenever — the core reconnects and re-negotiates capabilities.

A migration run that stops part-way is finished by running it again: each file applies whole or not at all, and a file that applied but was never recorded applies again harmlessly.

Rolling back a schema change is not supported. Migrations are append-only and there are no down-migrations, because a down-migration that drops a column drops the data in it, and the moment somebody wants one is the moment that data matters. Roll forward.

The recovery drill ​

make test-recovery runs the procedures on this page against real containers. It upgrades a v0.1.1 quickstart to the core built from this tree, and on the way starts the new core before migrating, interrupts a migration, rolls the binary back, restores a backup, starts with the wrong key, and gives up a lost one. Each check names the sentence on this page it proves, and fails when that sentence changes. CI runs it weekly, and on changes to the migrations, the store, the server's startup or the key handling.

Health and observability ​

Endpoints ​

GET /healthz                      the core is up. Says nothing about internals
GET /api/v1/gateways              per-gateway connection state and capabilities
POST /api/v1/ca-accounts/:id/health   ask one CA account's gateway to check its CA
GET /api/v1/dashboard/stats       counts, by status
GET /api/v1/events                the live stream (SSE)

/healthz is deliberately uninformative — it is a liveness probe, and a liveness probe that reports internals is an unauthenticated information leak.

What to watch ​

SignalWhereMeans
CAs in CRITICALGET /pki/authorities?status=CRITICALA CA is inside 30 days. Everything it signed dies with it
Renewal jobs FAILEDGET /api/v1/renewalsA CA is refusing, or a challenge is failing
Deployment jobs FAILEDGET /api/v1/deploymentsA certificate renewed and did not reach the server
verification_attempts climbingcertificate detailThe new certificate is not being served. The renewal is not done
Agents not reportingGET /api/v1/agentsA host has stopped; its certificates will expire silently
notification.failed in the audit logGET /dashboard/activity?actions=notification.failedAlerts are not reaching anyone

The last one deserves attention. A monitoring system whose alerting is broken is worse than none, because it is trusted.

Logs ​

Structured via log/slog. logging.format: json for ingestion.

The request logger never writes the query string, because display tokens travel in it. If you add your own access logging at the proxy, do the same.

Routine tasks ​

Connect a CA ​

These examples carry -b "$JAR", a cookie jar from signing in. There is no anonymous mode, so a command without a credential is a 401. See Authentication for the one-liner that fills it, or substitute -H "Authorization: Bearer $TOKEN".

bash
curl -b "$JAR" -X POST localhost:8080/api/v1/ca-accounts -H 'Content-Type: application/json' -d '{
  "name": "vault-issuing", "provider_type": "vault",
  "gateway_addr": "gateway-vault:9093",
  "config": {"address": "https://vault.internal:8200", "mount": "pki-int",
             "role": "web", "auth_method": "approle",
             "role_id": "...", "secret_id": "..."}}'

The configuration is validated against the CA before it is stored, and the response carries warnings worth reading — see gateways/vault.md. The CAs behind the account are imported at the same time.

Import the CAs behind an account ​

bash
curl -b "$JAR" -X POST localhost:8080/api/v1/pki/authorities/import
curl -b "$JAR" -X POST 'localhost:8080/api/v1/pki/authorities/import?account=vault-issuing'

Runs automatically every 12 hours and on account creation. This is for the moment after somebody has rotated an issuer.

Force a renewal ​

bash
curl -b "$JAR" -X POST localhost:8080/api/v1/certificates/$ID/renew

Enqueues a job; the queue runs it. The response is the job, not the certificate.

Check what is actually being served ​

bash
curl -b "$JAR" -X POST localhost:8080/api/v1/certificates/$ID/verify

Opens a TLS connection to the endpoint and compares the fingerprint being served against the one stored. This is the difference between "renewed" and "renewed and in use".

Revoke ​

bash
curl -b "$JAR" -X POST localhost:8080/api/v1/certificates/$ID/revoke \
  -H 'Content-Type: application/json' \
  -d '{"reason": 1}'

Admin only. The CA is told first, and the record changes only if the CA agreed. Every failure on this path — an unreachable gateway, a CA that declines, a certificate CertPilot holds no copy of — leaves the record exactly as it was and says so, because a row reading REVOKED beside a certificate that still answers handshakes is the one outcome worse than not revoking at all.

reason is required rather than defaulted. "Unspecified" is a legitimate answer, but it should be one somebody chose: the reason is what tells the next reader whether a key was compromised or a service was simply retired.

CodeReason
0unspecified
1keyCompromise
3affiliationChanged
4superseded
5cessationOfOperation
9privilegeWithdrawn

Anything else is a 400 that lists these. RFC 5280 defines more codes; CertPilot accepts the subset a CA will act on.

What can refuse it:

ResponseMeaning
400No reason, or one not in the table
400CertPilot holds no copy of the certificate, so it cannot be shown to a CA — usually a discovered certificate. Revoke it at the issuing CA
400Not bound to a CA account, so there is nowhere to send the revocation
404No such certificate
409Already revoked. The CA is not asked a second time
502The gateway is not connected, or the CA declined. Nothing has been changed

The one dangerous outcome is a 500 reading "the CA has revoked this certificate, but CertPilot could not record it". The certificate is genuinely dead; the record is wrong until the write succeeds. It is stated plainly rather than reported as a generic failure because the usual problem is the opposite way round.

Revocation is audited (certificate.revoked, naming the actor, reason and fingerprint) and published on the event stream as cert.revoked, so a wall display reflects it without waiting for a sweep.

Delete — not a substitute for revoking ​

DELETE /api/v1/certificates/:id deletes the record. It does not revoke.

It refuses with a 409 on a certificate that is still live, and names the revoke endpoint in the error. Expired and already-revoked certificates delete without argument: the first authenticates nothing, and for the second the CA has already been told.

?forget=true is the deliberate override, for a certificate you want CertPilot to stop tracking while it remains valid — an estate you no longer own, say. It is recorded in the audit log as the choice it is.

Take a CA out of the inventory ​

bash
curl -b "$JAR" -X DELETE localhost:8080/api/v1/pki/authorities/$ID

Admin only. Note that a gateway-imported CA will come back on the next import sweep if it is still offered by its mount — which is usually what you want. To stop tracking it, remove the CA account.