# Notifications

> Wire Slack, email, Telegram or a webhook so a failed backup is reported the moment its run ends.

Source: https://oddk.dev/docs/notifications/
Project: https://github.com/AndrianBdn/oddk


Configure this before you need it. A deployment with no notification channel has
no way to tell you it has stopped protecting itself — and the failure mode is not
a loud one. Scheduled captures keep running, keep failing, and keep reporting
nothing, until the morning you actually need a restore.

{{< callout type="warning" >}}
`oddk checklist` reports when no channel is configured. That warning is the one
thing standing between a silently broken schedule and a bad morning — treat it as
a to-do, not as noise.
{{< /callout >}}

## What sends you a message

| Trigger | When |
|---|---|
| **Degraded** | After `health.degraded_threshold` consecutive failed health checks (default **3**) |
| **Restored** | After `health.restored_threshold` consecutive good checks (default **2**) |
| **Run failed** | A scheduled snapshot or backup ending with any failed phase |
| **Recovered** | A clean scheduled run following a failed one |
| **Paused schedule** | Daily, and once at daemon start, while a schedule left paused by `snapshot apply` is still paused |

Two rules keep the channel readable, and both are deliberate:

**One message per run, never per phase.** A run where the capture, the upload
and both cleanups fail is *one* problem. Four messages for it is how an operator
learns to filter the channel — and a filtered channel is the same as no channel.

**One message per transition, not per check.** An instance parked in `error`
alerts once, not every 70 seconds. Clean-after-clean is silent.

## Configure a channel

Notifications are managed as a JSON array. Get a template,
edit it, apply it:

```bash
oddk notify help-add --type slack     # template for one type
oddk notify get > notifications.json  # current config, or a template if none
oddk notify apply --file notifications.json
```

The file is an array of `{name, type, config}` objects. Four types are supported.

{{< tabs >}}

  {{< tab name="Slack" >}}
The webhook URL **is** the credential — treat it like a password.

```json
[
  {
    "name": "ops-slack",
    "type": "slack",
    "config": { "slackWebhookUrl": "https://hooks.slack.com/services/T00/B00/xxxx" }
  }
]
```
  {{< /tab >}}

  {{< tab name="Email" >}}
```json
[
  {
    "name": "ops-email",
    "type": "email",
    "config": {
      "host": "smtp.example.com",
      "port": 587,
      "username": "oddk@example.com",
      "password": "…",
      "from": "oddk@example.com",
      "to": ["ops@example.com"],
      "startTls": true
    }
  }
]
```

Use `tls: true` for implicit TLS (port 465) or `startTls: true` to upgrade a plain
connection (port 587). `insecureSkipVerify` exists for internal relays with private
certificates — don't reach for it to make an error go away.
  {{< /tab >}}

  {{< tab name="Telegram" >}}
```json
[
  {
    "name": "ops-telegram",
    "type": "telegram",
    "config": { "token": "123456:ABC-DEF…", "chatId": "@my_ops_channel" }
  }
]
```

`chatId` accepts a numeric ID or an `@username`.
  {{< /tab >}}

  {{< tab name="Webhook" >}}
```json
[
  {
    "name": "ops-webhook",
    "type": "webhook",
    "config": {
      "url": "https://alerts.internal/hook",
      "headers": { "Authorization": "Bearer …" },
      "requestBodyType": "json",
      "requestBodyMessageKey": "text"
    }
  }
]
```

`requestBodyType` is `plain` (default), `post` (form-encoded) or `json`;
`requestBodyMessageKey` names the field the message goes into (default `text`).
  {{< /tab >}}

{{< /tabs >}}

## Prove it works

Configuring a channel is not the same as having one. Test it, and test it again
after any change to the receiving side:

```bash
oddk notify test        # send through every configured channel
oddk notify info        # overview of what is configured
oddk notify logs --limit 50
```

Health alerts get up to **three attempts per configured channel**. Scheduled-run
and paused-schedule messages get **one attempt per channel**. A notification
that fails to send is logged and **never** fails the task it was reporting on —
a broken Slack webhook must not turn a successful backup into a failed one.

## Tune the thresholds

```bash
oddk customkv set health.degraded_threshold.int --value 5
oddk customkv set health.restored_threshold.int --value 2
oddk customkv set health.check_interval_sec.int --value 70
oddk customkv set system.display.name.str --value "Production DB01"
```

`system.display.name.str` is what identifies the deployment in every message. Set
it if you run more than one host — otherwise every alert says the hostname, and
you will eventually act on the wrong machine.

## What the messages contain

A degraded notification names the affected instances **and why**:

```text
🚨 ODDK Service Degraded (Production DB01)

Time: 2026-08-27T03:14:07Z
Broken Instances: analytics
Details: instance_error:analytics

An instance in 'error' state is not being probed — it is recorded as broken.
That is either a failed operation (apply/switch/start) that needs repair, or a
configuration-only instance left behind by 'snapshot apply'. Run 'oddk checklist'.
```

A failed scheduled run names each failed phase, its cause, and — importantly —
which of three situations you are in, because they need different responses:

- **No new archive was produced.** Existing archives are untouched and retention
  keeps the newest regardless of age, so there is still something to restore
  from. This deployment is not gaining new protection until it is fixed.
- **An archive was produced but is not a complete capture.** The named instances
  hold no database contents and would restore as empty clusters. It was still
  kept, uploaded and catalogued, because it remains the newest restore point for
  every *other* instance.
- **The archive is fine but could not be shipped or pruned.** A later phase
  failed; the capture itself is good.

Then check the history directly:

```bash
oddk cron logs --failures
oddk cron logs --instance my-app
```

## One thing to know

**Egress is not restricted.** Webhook and Slack URLs are sent verbatim — there is
no allow-list on the target host. Anyone holding the API token can point a webhook
at an internal address and trigger it. This is accepted under oddk's threat model
(one trusted operator, one host, token-holder already has admin-equivalent
control), but it is a reason not to hand the token out casually.