Notifications

Configure this before you need it. A deployment with no notification channel has no way to tell you it has stopped protecting itself — and the failure mode is not a loud one. Scheduled captures keep running, keep failing, and keep reporting nothing, until the morning you actually need a restore.

What sends you a message

Two rules keep the channel readable, and both are deliberate:

One message per run, never per phase. A night where the capture, the upload and both cleanups fail is one problem. Four messages for it is how an operator learns to filter the channel — and a filtered channel is the same as no channel.

One message per transition, not per check. An instance parked in error alerts once, not every 70 seconds. Clean-after-clean is silent.

Configure a channel

Notifications are managed as a JSON array, like offsite settings. Get a template, edit it, apply it:

oddk notify help-add --type slack     # template for one type
oddk notify get > notifications.json  # current config, or a template if none
oddk notify apply --file notifications.json

The file is an array of {name, type, config} objects. Four types are supported.

[
{
  "name": "ops-slack",
  "type": "slack",
  "config": { "slackWebhookUrl": "https://hooks.slack.com/services/T00/B00/xxxx" }
}
]

[
{
  "name": "ops-email",
  "type": "email",
  "config": {
    "host": "smtp.example.com",
    "port": 587,
    "username": "oddk@example.com",
    "password": "…",
    "from": "oddk@example.com",
    "to": ["ops@example.com"],
    "startTls": true
  }
}
]

[
{
  "name": "ops-telegram",
  "type": "telegram",
  "config": { "token": "123456:ABC-DEF…", "chatId": "@my_ops_channel" }
}
]

[
{
  "name": "ops-webhook",
  "type": "webhook",
  "config": {
    "url": "https://alerts.internal/hook",
    "headers": { "Authorization": "Bearer …" },
    "requestBodyType": "json",
    "requestBodyMessageKey": "text"
  }
}
]

Prove it works

Configuring a channel is not the same as having one. Test it, and test it again after any change to the receiving side:

oddk notify test        # send through every configured channel
oddk notify info        # overview of what is configured
oddk notify logs --limit 50

Delivery is retried across every configured channel, three attempts each. A notification that fails to send is logged and never fails the task it was reporting on — a broken Slack webhook must not turn a successful backup into a failed one.

Tune the thresholds

oddk customkv set health.degraded_threshold.int --value 5
oddk customkv set health.restored_threshold.int --value 2
oddk customkv set health.check_interval_sec.int --value 70
oddk customkv set system.display.name.str --value "Production DB01"

system.display.name.str is what identifies the deployment in every message. Set it if you run more than one host — otherwise every alert says the hostname, and you will eventually act on the wrong machine.

What the messages contain

A degraded notification names the affected instances and why:

🚨 ODDK Service Degraded (Production DB01)

Time: 2026-08-27T03:14:07Z
Broken Instances: analytics
Details: instance_error:analytics

An instance in 'error' state is not being probed — it is recorded as broken.
That is either a failed operation (apply/switch/start) that needs repair, or a
configuration-only instance left behind by 'snapshot apply'. Run 'oddk checklist'.

A failed scheduled run names each failed phase, its cause, and — importantly — which of three situations you are in, because they need different responses:

• No new archive was produced. Existing archives are untouched and retention keeps the newest regardless of age, so there is still something to restore from. This deployment is not gaining new protection until it is fixed.
• An archive was produced but is not a complete capture. The named instances hold no database contents and would restore as empty clusters. It was still kept, uploaded and catalogued, because it remains the newest restore point for every other instance.
• The archive is fine but could not be shipped or pruned. A later phase failed; the capture itself is good.

Then check the history directly:

oddk cron logs --failures
oddk cron logs --instance my-app

Two things to know

Egress is not restricted. Webhook and Slack URLs are sent verbatim — there is no allow-list on the target host. Anyone holding the API token can point a webhook at an internal address and trigger it. This is accepted under oddk’s threat model (one trusted operator, one host, token-holder already has admin-equivalent control), but it is a reason not to hand the token out casually.

Rotate credentials that predate v0.1.80. Notification configs were stored as plain JSON before then, and oddk.db ships verbatim inside every snapshot archive — which is not encrypted and goes to S3. That column holds the credential for every channel: SMTP password, Slack webhook URL, Telegram bot token, webhook auth headers. They are encrypted now, but encrypting the column does not unpublish archives already written. If you ran an earlier version, rotate.