Backup and recovery

An offline, complete-instance recovery path for operators.

Requires the essentials update. The original downloadable preview has repository bundle exports, but no full-instance backup command. Build the latest source.

gitsvc backup create, verify and restore are offline operator commands, not a scheduler or remote-storage service. A Git bundle exports one repository; soft-delete restore requires the original disk. Neither replaces this backup.

Contents and boundaries

The archive contains the entire configured data_root: all repository bytes (including archived/deleted repositories), policy/users/keys/memberships, revoked fingerprint and route tombstones, audit, durable operations, retained uploads, staging and exports, volume marker, and any secrets already stored there. SQLite is captured with VACUUM INTO, which includes committed WAL transactions. The raw database/WAL/SHM and the live owner-lock file are excluded from the tree copy and replaced by the consistent database snapshot. No hooks execute.

The service TOML and configured private SMTP file are included automatically. The operator must inventory every additional file needed on this host:

• Every private SSH host key used by either Git listener, including shared port 22. Name individual key files with purpose ssh-host-key; include public key files too. At least one private OpenSSH key is required and must be mode 0600.
• Web configuration, private ACME cache or proxy certificate keys/configuration.
• Operator/bridge token files outside the data root and other required secrets.
• Dedicated SSH configuration, Git-only shared-SSH rules, systemd units/drop-ins, resource/PAM policy and host identity/group records needed for reconstruction. Record numeric UIDs/GIDs and the actual hard volume/quota boundary separately.

Example /secure/backup-inventory.json (adapt paths after inspecting the host):

{"paths":[
  {"path":"/etc/gitsvc","purpose":"configuration"},
  {"path":"/var/lib/gitsvc/ssh-host-keys/ssh_host_ed25519_key","purpose":"ssh-host-key"},
  {"path":"/etc/ssh/ssh_host_ed25519_key","purpose":"ssh-host-key"},
  {"path":"/etc/ssh/ssh_host_ed25519_key.pub","purpose":"ssh-host-public-key"},
  {"path":"/var/lib/gitsvc-web/acme","purpose":"tls-state"},
  {"path":"/secure/operator-token","purpose":"operator-token"}
]}

There is no automatic discovery of reverse-proxy dependencies or host SSH includes. Review backup verify's manifest against the host inventory; omission of an external secret cannot be detected by a repository service. Include only necessary regular files/directories. Symlinks, hard-linked files, devices, sockets and non-UTF-8 filesystem names fail backup safely. Recreate the fixed binary/hook symlinks from the reviewed package; do not inventory executable symlinks.

Excluded unless explicitly inventoried: OS/packages/binaries, DNS/firewall state, remote storage credentials, other applications, logs outside data_root, external snapshots, and all changes after the backup. Keep the matching reviewed binary, installation package, inventory and recovery encryption key off-server too. The initial format requires the exact service version and schema 5; it does not perform upgrades/downgrades. A backup contains old secrets and revocation state: reapply all revocations/denies since its timestamp before reopening access. A later purge does not erase copies in older backups.

Maintenance window and creation

1. Block new Git logins on both listeners without stopping administrative SSH. Drain operations while the daemon runs. Resolve pending/failed work as appropriate; pending and running work must finish before backup. Failed purge intents must also be resolved: a partially removed repository cannot pass restore fsck.

2. Stop the web process, daemon, dedicated Git listener and all Git workers, including shared-port SSH session scopes. Use the host-specific runbook; do not stop shared administrative SSH. Reconcile interrupted sessions offline if necessary. Keep configuration/certificate writers stopped too.

3. Run as root when required to read private host keys and preserve all owners:

   umask 077
   gitsvc --config /etc/gitsvc/config.toml backup create --workers-stopped \
     --inventory /secure/backup-inventory.json \
     --output /secure/backups/instance-2026-11-15.cpgbackup
   gitsvc backup verify /secure/backups/instance-2026-11-15.cpgbackup \
     > /secure/backups/instance-2026-11-15.manifest.json
   

   Copy code

   The ownership lock rejects a concurrent daemon/offline operator. The stop flag is the operator's assertion about external SSH workers and configuration writers; it cannot prove these processes stopped. Never supply it to bypass a maintenance window. Backup does not start or stop services itself.

4. Creation uses private temporary files, fsync and no-overwrite publication. It streams data with bounded buffers, verifies every checksum, then reports JSON status=completed. Provide space for the full uncompressed archive plus the SQLite snapshot outside data_root; no compression/deduplication is built in. Failed .partial files remain private for diagnosis and are never complete backups. Remove only proven task-owned temporaries after inspection.

5. Restart the daemon, verify readiness/storage, then web/Git access. See the checks below. Restart after a failed backup only once its cause is understood.

All commands return 0 only on success, 1 on validation/I/O/operational failure; CLI argument errors return 2. Preserve the exit status in external scheduling. A final archive is mode 0600; manifests reveal paths/identity metadata and must also remain private. Checksums detect corruption, not malicious replacement.

Encryption, off-server copy and retention

Use existing encryption and transfer tools. Example with an existing GPG recovery recipient (verify its fingerprint separately; keep its private key off-server):

umask 077
gpg --output instance.cpgbackup.gpg --encrypt --recipient RECOVERY_FINGERPRINT \
  /secure/backups/instance-2026-11-15.cpgbackup
sha256sum instance.cpgbackup.gpg > instance.cpgbackup.gpg.sha256
scp -p instance.cpgbackup.gpg instance.cpgbackup.gpg.sha256 backup-host:private/cpgit/

Use pinned SSH host identity, restricted backup-host access and encrypted storage; never upload plaintext archives or put secrets in URLs/argv/logs. Verify the remote checksum, retrieval, decryption and restore before calling the backup recoverable. Keep encryption keys/passphrases separate from archives. An existing host timer/cron can invoke the maintenance script and alert on any nonzero exit; use its existing facilities, with explicit restart/error handling. Do not schedule unattended stops of shared administrative SSH.

Choose retention based on recovery needs and secret exposure; a practical starting policy is seven daily and four weekly verified encrypted copies, with one offline copy. Implement retention with the existing storage tool, never repository purge or Git pruning. Keep the last known restorable copy until a replacement is verified. Record backup timestamp, archive checksum, remote verification result, exit status and last successful restore rehearsal outside the lost instance.

Restore on an empty replacement

Keep replacement endpoints closed and all service/web/Git workers stopped. Install the matching reviewed binary and patched native Git/OpenSSH; do not initialize a new policy store. Recover using only off-server material:

umask 077
scp -p backup-host:private/cpgit/instance.cpgbackup.gpg .
scp -p backup-host:private/cpgit/instance.cpgbackup.gpg.sha256 .
sha256sum --check instance.cpgbackup.gpg.sha256
gpg --output instance.cpgbackup --decrypt instance.cpgbackup.gpg
chmod 600 instance.cpgbackup
gitsvc backup verify "$PWD/instance.cpgbackup" > verified-manifest.json
sudo gitsvc backup restore "$PWD/instance.cpgbackup" /srv/recovery --workers-stopped

/srv/recovery must not exist, even as an empty directory or symlink. Its parent must exist. Restore verifies format/schema, manifest and all file hashes, paths and ownership feasibility before creating staging. It checks SQLite integrity/foreign keys and native full strict fsck of active/archived/deleted repositories before publishing the tree. Invalid input cannot overwrite an existing destination. I/O or native-validation failure retains a private staging tree and returns failure; it never starts services or exposes partial data.

The result is data/, system/<original absolute path without leading slash>, and a private recovery-manifest.json. Numeric ownership/modes are preserved; root is required when archived owners differ from the invoking account. The outer directory remains 0700. Browser sessions and pending login codes are invalidated with an audit event; users sign in again. SSH keys, API token hashes, permissions, denies and tombstones retain their snapshot policy.

Review the recovered host files, then install them deliberately on the empty replacement. Never recursively copy system/ over / or overwrite another host's administrative SSH policy. Recreate recorded UIDs/GIDs, install the data on the intended empty bounded volume, restore its marker, validate actual volume/ project quota, install configuration/secrets at their configured paths with the recorded ownership, and recreate trusted fixed hooks from the package. Adapt data_root, runtime/binary paths and UID/GID settings only if the replacement layout requires it; update matching repository ownership consistently. Do not retain storage_boundary_confirmed=true without verifying the new boundary. Certificate/SSH host identities must match the inventory. Restoring shared SSH host keys affects that replacement host's administrative identity too.

Start the daemon with Git/web access still closed. Check gitsvc status and /v1/storage, inspect failed operations/audit, and reapply later revocations. Validate sshd -t and effective Git-only policy, UID/socket separation and worker resource limits. Check full ref maps/history, fresh clone and revoked-key denial on an explicitly disposable test repository; confirm HTTPS/session sign-in and private management isolation. Open access only after these pass. Endpoint/DNS and client-remote changes require their own authorization. A rootless local rehearsal cannot establish replacement-host isolation, quotas or public TLS.

Verified off-server recovery, 15 November 2026

The deployed essentials build produced a full production backup with 1,722 entries and 2,615,604,983 bytes. Its SHA-256 was dc91dd80a831423ed8b060eb7521122de0c4ae8ab91f220866cfb93d725128fd. The archive was encrypted, transferred off the Git server to a separate host, and restored into a fresh, network-isolated offline recovery tree. Native full Git fsck and SQLite validation passed.

This advances the evidence beyond the earlier encrypted loopback rehearsal. It verifies the production backup, off-server transfer and offline restoration. It does not establish a public replacement-host boot/cutover, recurring backups or retention. Those require deliberate setup and acceptance using the procedures above. Keep encryption keys separate and validate the target volume, identities and current revocations before reopening access.