Self-hosted Postgres,
built for the night
it all goes wrong.
oddk gives one Linux box the ergonomics of a managed database service — instances, whole-deployment snapshots, offsite copies, health alerts. The difference is what happens at 3 a.m.: recovery is the part this tool was actually built around, and the parts below are the failures it was hardened against.
$ curl -fsSL https://oddk.dev/install | sh
Fig. 1 — Provisioning
One command
to a live database.
Pick cores, memory and a port. oddk pulls the image, sizes shared memory against that instance's RAM, wires the bridge network, generates and encrypts a password — and blocks until Postgres actually answers. If readiness is never reached it rolls the container and volume back rather than handing you a broken row.
$ oddk create --name app --version 17 --port 5432 --cpu 4 --ram 8 pulling postgres:17 ......................... done creating volume oddk-data-app ............... done waiting for postgres to accept connections .. done instance "app" running $ oddk instance get-postgres-password app --conn postgresql://postgres:••••••••••••@10.88.0.1:5432/postgres $ oddk list NAME VERSION PORT CPU RAM STATUS app 17 5432 4 8 GB running
Fig. 2 — Failure catalogue
Every one of these
reported success.
A backup tool is only worth what it does on the day you need it, and the dangerous failures are not the loud ones — they are the ones that return zero and log nothing. These are real defects found in oddk, what each one silently did, and what it now does instead. They are the reason the design is shaped the way it is.
The capture that archived nothing
Snapshot capture trusted the instance's stored status. That string drifts —
a crashed switch leaves its transient status behind, a single failed probe latches
broken-port onto a server that is up and serving.
So a healthy, serving cluster was reduced to configuration-only in every subsequent archive — while the snapshot reported success, was catalogued, uploaded, and counted toward the retention floor.
A degraded entry is reported as a failure at every layer: manifest, catalogue, checklist, notification, and a non-zero exit — while the archive is still kept and shipped.
The corrupt archive that restored clean
A tar reader stops at the tar terminator — before the end of the compressed frame.
So the checksum that would catch corruption was never read, and extraction returned
err == nil on damaged archives.
Restores did not discover this. They completed, with different data.
zstd frame drain 40 / 40 caught
Archives are now verified on the way out and on the way in — before a catalogue row claims a copy, and before any restore is allowed to depend on the file.
Retention deleting the last restore point
Retention runs on a schedule, including on nights the capture failed. With an age-only rule, a job that had been failing longer than the retention window would expire every archive it had — leaving nothing to restore from, at precisely the moment the deployment was least able to make a new one.
The floor counts surviving copies, not catalogue rows: a row whose file was deleted out of band protects nothing, and letting it hold a slot meant guarding ghosts while deleting real archives.
The DR rehearsal that ate production's backups
A snapshot carries the source host's configuration — including its offsite bucket and credentials. Restore it onto a second machine while the first is still live, and that machine starts uploading into the same bucket and running retention against it.
It deletes objects the source still catalogues. Silently, and to the wrong host's archives.
snapshot apply restores every schedule paused,
never removed — a paused plan keeps its settings, resumes with one command, and is
loud: the checklist prints it, and a daily notice repeats until it is dealt with.Removing them instead would trade a rehearsal hazard for a worse one — a DR host that protects nothing and never says so.
Credentials riding in the clear
Notification settings were stored as plain JSON. That column holds the credential for every channel: SMTP password, Slack webhook URL (the URL is the credential), Telegram bot token, arbitrary webhook auth headers.
That database is embedded verbatim in every snapshot, and snapshots go to S3.
Encrypting the column does not unpublish what already shipped. Archives written earlier still contain them. The changelog says so, and says to rotate.
Fig. 3 — Recovery procedure
Rebuilding a host
that no longer exists.
One archive plus one key. This runs locally against the data directory rather than through the daemon — deliberately, because the daemon is the thing that died.
Preflight, cheapest first
Version, daemon stopped, deployment empty, images present, archive extracted, master key decrypts, host resources, locale providers, ports. Every refusal ends “No ODDK state on this host was modified.”
Images pulled before anything destructive
A fresh DR host has an empty image cache. Discovering that after the store was replaced would abort an apply that then cannot be retried.
Clusters restored and proven
Physical entries stream into the container before its first start. A cold-captured instance is started once to prove the restore, then stopped again to match how it was captured.
Schedules restored paused
See dossier 04. Resume with snapshot setup-cron --resume once this
host owns its bucket.
# the daemon can't start. that's the point. $ sudo -u oddk oddk snapshot apply \ --s3-uri 's3://bk/*snapshots*/2026-08-27/s.tar.zst' \ --master-key /mnt/restore/master.key preflight: version .......... ok preflight: daemon stopped ... ok preflight: deployment empty . ok preflight: images ........... pulled 2 preflight: master key ....... decrypts preflight: host resources ... ok no state modified yet restoring app ............... ok restoring billing ........... ok restoring analytics ......... ok schedules restored PAUSED resume once this host owns its bucket
Fig. 4 — Out of scope
What oddk is not.
Knowing where a tool stops is worth more than a longer feature list. If you need any of these, oddk is the wrong choice and will stay the wrong choice.
Open the case.
Installing and updating are the same command. About ten seconds.
$ curl -fsSL https://oddk.dev/install | sh