# Running TelltaleDNS > **Status:** stable releases are listed on the [releases page](https://github.com/paimonsoror/telltaledns/releases). This page describes `main`; each release's notes say what it contains. Releases before 1.0 can still change configuration keys and APIs, and their notes call out every such change. What stands between 0.x and 1.0 is tracked in [`docs/v1-gate.md`](v1-gate.md). ## Start ```sh telltale run # uses $TELLTALE_CONFIG, else /etc/telltale/telltale.toml, else defaults telltale run -c telltale.toml # explicit config file(s); later files override earlier ones ``` A minimal working config: ```toml [[listen]] proto = "udp" addr = "127.0.0.1:5300" [[listen]] proto = "tcp" addr = "127.0.0.1:5300" [[upstream]] name = "cloudflare" url = "udp://1.1.1.1" [[upstream]] name = "quad9" url = "udp://9.9.9.9" [[upstream_group]] name = "default" # queries that match no route use this group members = ["cloudflare", "quad9"] strategy = "fastest" # failover | round_robin | weighted | fastest | parallel ``` ```sh telltale run -c dev.toml dig @127.0.0.1 -p 5300 example.com ``` Defaults without `[[listen]]`: UDP and TCP port 53 on `0.0.0.0` and `[::]`, with one worker thread per available CPU (container CPU limits are respected). Ports below 1024 need root or `CAP_NET_BIND_SERVICE`. DoT, DoH, DoQ, and DoH3 listeners are described in "[Encrypted DNS for your devices](#encrypted-dns-for-your-devices-dot-doh-and-doq)". ## Container image `ghcr.io/paimonsoror/telltale:latest` is the newest stable release (also `:X.Y.Z`, `:X.Y`, and `:X`; `:edge` is built from every commit on `main`), for `linux/amd64`, `linux/arm64` (Raspberry Pi 3/4/5 with a 64-bit OS), and `linux/arm/v7` (32-bit Pi OS). It contains one static binary, CA certificates, and time-zone data, about 13 MiB compressed, with no shell. It runs as user `65532:65532`. ```sh docker run -d --name telltale --restart unless-stopped \ --read-only --cap-drop ALL --security-opt no-new-privileges \ -p 53:53/udp -p 53:53/tcp -p 8053:8053 -p 9153:9153 \ -v "$PWD/telltale.toml:/etc/telltale/telltale.toml:ro" \ -v telltale-data:/var/lib/telltale \ ghcr.io/paimonsoror/telltale:latest ``` - The config is read from `/etc/telltale/telltale.toml` when it exists; otherwise the built-in defaults apply. `/var/lib/telltale` is the only path written to. - With `-p` (bridge networking), port 53 works with every capability dropped: Docker lets unprivileged users bind low ports inside the container's own network namespace. - With `--network host` (useful on a Pi, so the logs show real client addresses), that exception doesn't apply. Let unprivileged users bind port 53 on the host: ```sh echo 'net.ipv4.ip_unprivileged_port_start=53' | sudo tee /etc/sysctl.d/50-telltale.conf && sudo sysctl --system ``` Alternatively, run the container as root with only that one capability: `--user 0:0 --cap-drop ALL --cap-add NET_BIND_SERVICE`. Adding `NET_BIND_SERVICE` while running as `65532` doesn't work, because Docker doesn't pass added capabilities to non-root users. - Reload with `docker kill -s HUP telltale`; stop with `docker stop` (a graceful drain, see [Stop](#stop)). Build it yourself with `docker buildx build -t telltale:dev .` (add `--platform linux/amd64,linux/arm64,linux/arm/v7` for all three). The build cross-compiles, so it doesn't need QEMU. ## Raspberry Pi and Linux Two ways, both with real client addresses (host networking) and port 53 without running as a full root process. Pi 3, 4, 5, and Zero 2 W are supported (64-bit or 32-bit OS); ARMv6 boards (Pi 1, Zero W) aren't. **Docker Compose (recommended):** `deploy/compose/` has `compose.yaml` and a starter `telltale.toml` (encrypted upstreams, one balanced blocklist, a 7-day query log). ```sh mkdir telltale && cd telltale curl -fsSLO https://raw.githubusercontent.com/paimonsoror/telltaledns/main/deploy/compose/compose.yaml curl -fsSLO https://raw.githubusercontent.com/paimonsoror/telltaledns/main/deploy/compose/telltale.toml docker compose up -d docker compose exec telltale telltale auth setup-token # then open http://:8053/ ``` The container uses the host network, a read-only root filesystem, and only `NET_BIND_SERVICE`, as root inside the container by default. To run it as the image's unprivileged user, set `net.ipv4.ip_unprivileged_port_start=53` on the host (`echo net.ipv4.ip_unprivileged_port_start=53 | sudo tee /etc/sysctl.d/50-telltale.conf && sudo sysctl --system`) and switch `user:` in `compose.yaml`. Updates: `docker compose pull && docker compose up -d`. Reload the config without a restart: `docker compose kill -s HUP`. **Native (systemd):** `deploy/systemd/install.sh` downloads the static binary for your CPU, checks the release signature (minisign) and its SHA-256, creates a `telltale` system user, installs a sandboxed systemd unit (`ProtectSystem=strict`, only `CAP_NET_BIND_SERVICE`, `MemoryMax=256M`) and the starter config (an existing `/etc/telltale/telltale.toml` is kept), then starts the service and waits until it's ready. ```sh curl -fsSLO https://raw.githubusercontent.com/paimonsoror/telltaledns/main/deploy/systemd/install.sh less install.sh # read before running as root sudo sh install.sh # the latest release; --edge: builds from main sudo -u telltale telltale auth setup-token -c /etc/telltale/telltale.toml ``` **Guided setup.** Run in a terminal on a new machine, the installer asks one question first: press Enter for the recommended settings (the starter config above), or choose **Walk me through the options**. Each step shows its default in `[brackets]`; Enter or `skip` keeps it: 1. **Builds:** releases or edge. 2. **Upstreams:** Cloudflare and Quad9 (encrypted, the default), one provider (Cloudflare, Quad9, Google, Mullvad, AdGuard), plain DNS for networks that block port 853, or your own server. 3. **Starting blocklists:** one or more of HaGeZi Pro (the default), Light, Pro++, Threat Intelligence, Steven Black's hosts, and OISD small, or none (add lists later on the Lists page). 4. **Where DNS listens:** every address, or one. 5. **systemd-resolved:** turn its stub listener off if it holds port 53. 6. **The first admin:** a setup token for the web UI (the default), or a username and password now. 7. **The query log:** how many days to keep, and how much it records (privacy level 0 to 3). A summary comes before anything is installed. The answers become `/etc/telltale/telltale.toml`, checked with `telltale config check` before the service starts. - **No questions** when stdin isn't a terminal (`curl … | sh`, automation), with `--yes`, or when `/etc/telltale/telltale.toml` already exists (re-runs only upgrade). To see the questions again on a machine that already has TelltaleDNS, run `sh install.sh --config-only /tmp/test.toml` (it changes nothing), or uninstall first (below). - **Automation:** `--interactive` asks even without a terminal and reads the answers from stdin; `--config-only FILE` writes the config and stops (no root needed), to see what the answers produce. - **The first admin "now"** is created on the service's first start from an Argon2id hash in a root-only systemd drop-in, which the installer removes once the service is up. The password itself isn't stored. Re-running it upgrades the binary in place. `sudo telltale self-update --restart` does the same check-and-swap from the binary itself (`--channel edge` for builds from main, `--check` to only look): it verifies the signature with the release key built into the binary and the binary's SHA-256, test-runs the download, swaps it in atomically, and keeps the previous binary as `telltale.old`. In containers it refuses (pull a new image instead). **Uninstalling:** `sudo sh install.sh --uninstall` stops and disables the service and removes its unit and the binary (`telltale.old` too). - **The system resolver:** if the install turned off systemd-resolved's stub listener, it asks to restore it (the default). `/etc/resolv.conf` is put back as the install found it (the install records the link it replaced, or keeps a copy of the file), and systemd-resolved is restarted, so the machine resolves names again without TelltaleDNS. Without a terminal or with `--yes` it restores without asking; `--keep-resolver` leaves it as the install set it. Installs from before this was recorded get the usual stub link (`/run/systemd/resolve/stub-resolv.conf`). - **Config and data** (`/etc/telltale`, `/var/lib/telltale`: users, query log, lists) are kept, so a later install picks them up. It asks whether to delete them; `--purge` deletes them and the `telltale` user without asking. - Point your router's DNS setting back at your previous server if it handed out this machine. **Port 53 already in use:** on Ubuntu and some Debian setups, systemd-resolved's stub listener holds `127.0.0.53:53`. `install.sh` offers to turn it off (`--disable-resolved-stub` does it without asking: the machine then resolves through TelltaleDNS); for Compose, add `DNSStubListener=no` under `[Resolve]` in `/etc/systemd/resolved.conf.d/telltale.conf` and `sudo systemctl restart systemd-resolved`. Alternatively listen on the LAN address only with `[[listen]]`. Pi-hole or another resolver on the same machine must be stopped first. **Small Pis and SD cards:** the query log writes continuously; keep `[telemetry.qlog] retention_days` short (the starter config uses 7) or put `/var/lib/telltale` on a USB SSD. On a Pi Zero 2 W (512 MB) also set `[node] workers = 2` and `[cache] max_bytes = "16MiB"`, and prefer smaller blocklists. Then point your router's DHCP DNS server setting at the Pi's address so every device uses it. ## Moving from Pi-hole `telltale import pihole PATH -o pihole.toml` turns a Pi-hole setup into TelltaleDNS configuration. PATH is a Teleporter export (Settings → Teleporter → Export: a `.zip` from Pi-hole v6, a `.tar.gz` from v5), a `gravity.db`, or a copy of `/etc/pihole`. The output is checked to load before it's written. | Pi-hole | TelltaleDNS | |---|---| | Upstream DNS servers (`IP#port`) | `[[upstream]]` `pihole-N` in the `default` upstream group, strategy `fastest` (Pi-hole also prefers the fastest server) | | Conditional forwarding (rev servers) | an upstream, an upstream group, and a `[[route]]` for the local domain and every reverse zone of the network (a /23 is two /24 zones), with a DNSSEC negative trust anchor | | Local DNS records, CNAME records | `[[record]]` A/AAAA and CNAME (with the CNAME's TTL) | | Adlists (`https://`, `file://`) | `[[list]]` named `pihole--`, with `url` or `path`; switched-off ones stay with `enabled = false`; v6 allowlists become `kind = "allow"` | | Allowed / denied domains, exact and regex | inline lists `pihole-allow-exact`, `pihole-deny-regex`, ..., one per set of groups. Exact entries match only that name (`match = "exact"`). Regexes keep `;querytype=` and `;invert` | | Groups (Default → `default`) | `[[group]]` with exactly the lists Pi-hole gave it. A switched-off group gets no lists | | Clients (IP, CIDR, MAC) | `[[client]]` named by the Pi-hole comment, in the same groups. Clients in no group (nothing is blocked for them in Pi-hole) go to `pihole-no-group`, which has no lists | | DHCP reservations | named devices: the MAC and IP join a matching client, or become a new one named by the host name | | DNSSEC, rate limit, blocking mode (NULL, NXDOMAIN, NODATA) | `[dnssec] mode = "validate"`, `[ratelimit]`, each group's `block_mode` | The header lists what has no equivalent, by setting name (values, such as password hashes, are never copied): - clients matched by host name or network interface (add their IP or MAC instead); - the DHCP server (TelltaleDNS leaves DHCP to your router); - privacy level, web-server, and other Pi-hole-only settings. From v6, only settings marked as changed from their defaults are listed; - the `IP` blocking modes, entries switched off, and domain entries our regex engine can't run (backreferences and lookaround). Query history, the admin password, and API tokens aren't imported. The output is a complete starting configuration: on a fresh install, use it as is next to your node settings (`telltale run -c telltale.toml -c pihole.toml`). Into an existing one, merge by hand: both define the `default` upstream group and group. Upstreams on the Pi-hole machine itself (`127.0.0.1#5335`, usually unbound) are flagged: keep that resolver running next to TelltaleDNS (a built-in recursive resolver is on the post-1.0 list). CI imports real exports from the official Pi-hole v6 and v5 images and checks the answers (`deploy/pihole-import-e2e.sh`). ## Moving from Technitium `telltale import technitium http://192.168.1.2:5380 -o technitium.toml` reads a running Technitium DNS Server through its API and writes TelltaleDNS configuration. In Technitium, create an API token (Administration → Sessions → Create Token). Pass it in `TECHNITIUM_TOKEN`, or in a file with `--token-file`; it's never a command-line argument, so it doesn't show up in `ps`. The output is checked to load before it's written, and the token never appears in it. | Technitium | TelltaleDNS | |---|---| | Forwarders (UDP, TCP, TLS, HTTPS; `name (ip:port)` pins the address) | `[[upstream]]` `technitium-N` in the `default` group; concurrent forwarding becomes `parallel` with the same fan-out | | Primary zones | `[[record]]` A, AAAA, CNAME, PTR, TXT, MX, SRV, with their TTLs (SOA and apex NS aren't needed) | | Forwarder zones (conditional forwarding) | upstreams by priority, a group, and a `[[route]]` for the zone. A negative trust anchor applies unless every forwarder of the zone validated | | Block list URLs (`!` = allow list), blocked and allowed zones | `[[list]]`; the zones become inline lists `technitium-blocked` / `technitium-allowed` (subtree, like Technitium) | | Blocking type (NXDOMAIN, custom address) | each group's `block_mode` | | Advanced Blocking app groups | `[[group]]` with the group's URL lists (shared when groups use the same URL), names, and regexes. The group for `0.0.0.0/0` becomes `default`; other networks become the group's `networks`. Server-wide lists apply to every group | | Blocking bypass list | a `technitium-bypass` group with those networks and no lists | | DHCP reservations | named devices (MAC and IP) | | DNSSEC validation, per-address rate limit, saving the cache | `[dnssec]`, `[ratelimit]` (queries per minute), `[cache] persist` | The header lists what isn't imported: - secondary and stub zones; - forwarders given only by name; - record types we don't serve locally; - groups chosen by listener or DoH host name; - per-list blocking answers; - the DHCP server; - encrypted listeners (set them up under Encrypted DNS for your devices); - recursion ACLs, TSIG, zone transfers, the outgoing proxy, and other apps. Technitium's backup files use a private binary format, so the importer reads the documented API instead (ADR-062). CI runs it against the official Technitium image (`deploy/technitium-import-e2e.sh`). ## Versions and updates Every build says exactly what it is, the same way on every architecture: ``` $ telltale --version telltale 0.1.0-edge.64 (commit 3f46fd4, built 2026-10-05T12:00:00Z, channel edge, aarch64-unknown-linux-musl) ``` **Version formats:** - releases are `0.1.0`; - builds of `main` are `0.1.0-edge.`; - a local build is `dev` with its commit. The same build identity, and how the node was installed (`native`, `container`, or `helm`), shows in the web UI's sidebar (with the node's name and, in a cluster, whether it's the **primary** or a **replica**: changes saved on a replica's UI go to the primary) and **Settings → System → Version and updates**, in `GET /api/v1/system/info` (`build`, `update`), in `telltale_build_info{version,commit,channel,target}`, and on the Cluster page for every node. **Update checks:** - **How:** once a day, a node reads its channel's `releases.json` (published with every release, signed with the release key) and compares versions: `up_to_date`, `available`, `newer` (a dev build), or `unknown` (not checked yet, or failing). An index whose signature doesn't verify is ignored. - **When:** a minute after the node starts, then every 24 hours (an hour after a failed check). To look now, an admin clicks **Check now** under **Settings → System → Version and updates** (`POST /api/v1/system/update-check`). It's handy on the edge channel, which publishes several builds a day. Clicks less than a minute apart show the last result without fetching again. Nothing is installed. - **Where it shows:** an **update** pill next to the version in the sidebar, the Version and updates panel (with the step for your install), the Cluster page (each node's version), and `telltale_update_available` (1 when an update exists). - **Air-gapped:** `[updates] check = false` means nothing leaves the node; `[updates] index_url` points at a mirror (still verified with the built-in key). **Checking what you run** (NFR-005): - **Binaries:** `SHA256SUMS` is signed with the release key (minisign; `install.sh` and `telltale self-update` check it for you). The public key is `deploy/release/telltale-release.pub`. - **Images** are signed with [cosign](https://docs.sigstore.dev/) through GitHub's keyless signing: the signature names the repository's image workflow, and is in Sigstore's public log. ```sh cosign verify ghcr.io/paimonsoror/telltale:0.5.0 \ --certificate-identity-regexp '^https://github\.com/paimonsoror/telltaledns/\.github/workflows/image\.yml@' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com ``` - **What's inside:** each binary carries its dependency list (`cargo audit bin telltale` reads it), and every release has a CycloneDX SBOM per binary (`telltale--linux.cdx.json`, listed in the signed `SHA256SUMS`). - **Rebuilding:** a release builds the same bytes from the same commit. The build date in `--version` is the commit's time. To compare, build the Dockerfile's `bin` stage with the build arguments `--version` shows (`TELLTALE_BUILD_VERSION`, `TELLTALE_BUILD_COMMIT`, `TELLTALE_BUILD_DATE`, `TELLTALE_BUILD_CHANNEL`) and `SOURCE_DATE_EPOCH` set to the commit's Unix time, then compare checksums. A nightly job builds every commit twice from scratch and fails if the two differ. TelltaleDNS never updates itself from the UI. The panel shows the step for your install: - **Native:** `sudo telltale self-update --restart` (`--channel edge` for edge). - **Container:** `docker compose pull && docker compose up -d`. - **Helm:** `helm upgrade`, or bump the chart version in your GitOps values. In a cluster, upgrade replicas first. ## Backups and moving to a new machine ```sh telltale backup create -o pi.ttbk # safe while TelltaleDNS runs telltale backup create --include-qlog -o pi.ttbk # also the query log (can be large) telltale backup show pi.ttbk # check it and list what's in it telltale backup restore pi.ttbk # on the new machine, TelltaleDNS stopped ``` Admins can also download one in the web UI (**Settings → System → Download a backup**) or with `GET /api/v1/backup`. Each download is recorded in the audit log as `backup.create`. Downloads leave out the query log; use the command for that. **In a backup:** - the config files the node was started with; - users (password hashes, two-factor secrets) and API tokens; - devices, names, and forwarded domains made in the UI; - the audit log, statistics history, and anomaly baselines; - the query log, with `--include-qlog`. **Left out:** - sign-in sessions (everyone signs in again); - downloaded lists, compiled snapshots, and the cache, which are rebuilt; - the cluster identity and its CA key. A restored cluster member starts on its own: run `telltale cluster init` or join it again. The file is zstd-compressed tar with a manifest of BLAKE3 checksums, written owner-only (`0600`); keep it private. **Restore:** - Every file is checked before anything is written, so a damaged or truncated backup changes nothing. - Existing files are only replaced with `--force`. - Data goes back to the directory it came from, or `--data-dir`. Config files go back where they were, or `--config-dir`. - It refuses while TelltaleDNS is running on that data directory (it can't take the directory's lock): stop the service first. - Files are given the data directory's owner, so a restore run as root works for the `telltale` service user. - If you restore into a different data directory, the restored config still names the old one: restore says so, and you set `data_dir` (or add a small file that does). ## Kubernetes (Helm) The chart is published with every release as an OCI artifact (k8s 1.26+, amd64 and arm64). Use a release version, or `0.1.0-edge.` builds that follow `main`. Its source is in `deploy/helm/telltale`, and `helm install telltale deploy/helm/telltale` from a checkout works too: ```sh helm install telltale oci://ghcr.io/paimonsoror/charts/telltale --version \ -n telltale --create-namespace \ --set service.dns.annotations."metallb\.universe\.tf/loadBalancerIPs"=192.168.1.53 kubectl -n telltale get svc telltale-dns # EXTERNAL-IP: point clients (DHCP) here ``` **Every option** is in the chart's commented values file: `helm show values oci://ghcr.io/paimonsoror/charts/telltale --version > values.yaml` (or [`deploy/helm/telltale/values.yaml`](../deploy/helm/telltale/values.yaml); the site's [Helm values](https://paimonsoror.github.io/telltaledns/helm-values.html) page lists each key with its default). Keep only what you change in your own file and install with `-f my-values.yaml`; the chart's schema rejects a mistyped value with its path. - **One LoadBalancer Service, UDP and TCP 53**, with `externalTrafficPolicy: Local` so the query log and per-device rules see real client addresses (verified in CI: a query through the LoadBalancer is logged with the sender's address). `Cluster` hides clients behind node addresses, and the install notes warn about it. Use your LB's annotation for a fixed IP: MetalLB `metallb.universe.tf/loadBalancerIPs`, Cilium `io.cilium/lb-ipam-ips`, kube-vip `kube-vip.io/loadbalancerIPs`. - **`hostNetwork: true`** answers on port 53 of the node itself (no LoadBalancer, MAC addresses visible). Binding 53 there needs root with only `NET_BIND_SERVICE` kept; on Ubuntu nodes turn off systemd-resolved's stub listener first. k3s's CoreDNS doesn't conflict. - **The UI and API**: `service.api.type: LoadBalancer`, or `ingress.enabled` with `ingress.host` (set `[api] trusted_proxies` in `config` to your ingress's network). - **Configuration**: `config:` is merged after the chart's own settings (listeners, data directory, ports): upstreams, lists, groups, `[auth.oidc]`, everything in `docs/configuration.md`. - **Secrets**: `auth.bootstrapAdmin.existingSecret` creates the first admin (keys `username` and `password` or `password-hash`); `secretMounts` mounts Secrets at `/etc/telltale-secrets/` (for example an OIDC `client_secret_file`). - **Monitoring**: `serviceMonitor.enabled`, `prometheusRule.enabled` (down, all upstreams down, SERVFAIL rate, stale lists, dropped telemetry, masked client IPs, device anomalies), and `grafanaDashboard.enabled` (a ConfigMap for the Grafana sidecar). - **`networkPolicy.enabled`** limits who may query (`dnsFrom`), reach the UI (`apiFrom`), and scrape (`metricsFrom`). - **Encrypted DNS**: `encrypted.dot.enabled` (853), `encrypted.doh.enabled` (443, plus `encrypted.doh.http3`), and `encrypted.doq.enabled` add DoT, DoH, and DoQ to the DNS Service; the certificate comes from `encrypted.tls.secretName` or a cert-manager `Certificate` (`encrypted.tls.certManager.issuerRef` and `dnsNames`; add a wildcard for client IDs) and is reloaded when renewed. `encrypted.proxyProtocol` accepts PROXY protocol v2 on TCP, DoT, and DoH. - **One replica or many.** `mode: allInOne` (the default) is one StatefulSet with a volume for lists, the query log, and users. **`mode: scaled`** adds resolver pods: ```sh helm install telltale deploy/helm/telltale -n telltale --set mode=scaled --set resolvers.replicas=3 ``` - The StatefulSet becomes the cluster's **controller**: it creates the cluster on first start, keeps the volume, and serves the UI and API. - Each **resolver pod** joins it automatically with a Secret the chart generates (48 random characters; with `cluster.bootstrapSecret.existingSecret`, use at least 32, or the log warns). It takes the controller's configuration and compiled lists, and ships its query log to the controller's volume. - Resolver pods are trusted less than the controller: they never get the cluster's signing key, can't vote or be promoted, can't step the primary down, and can't change anything on other nodes (forward configuration changes, pause blocking, or flush caches). Make those changes on the controller. - A resolver pod reports ready only once it has the configuration, so it never answers unfiltered. Pods come and go: one that's gone for `resolvers.ephemeralTtlSeconds` (default 600) leaves the cluster's list of nodes. The Cluster page shows them grouped by site ("k8s: 3 resolver pods"). - The DNS Service spreads queries over the controller and every resolver pod. - **A node outside Kubernetes** (a Pi) can join the same cluster: - expose the cluster port with `cluster.service.type: LoadBalancer`; - list that address in `cluster.advertise` (e.g. `["https://192.168.5.100:9443"]`); - create a token on the controller (`kubectl exec sts/telltale -- telltale cluster token create --url https://192.168.5.100:9443 -c /etc/telltale/00-chart.toml -c /etc/telltale/10-values.toml -c /etc/telltale/20-controller.toml`); - then run `telltale cluster join ` on the Pi. - **Staged rollouts** (off by default): `cluster.rollout.canaries: [ephemeral]` gives each change to the resolver pods (with the controller) first and to a node outside Kubernetes after a 5-minute bake, pinning the cluster back if the pods start failing. See [Rolling a change out in stages](#rolling-a-change-out-in-stages). - With Argo CD (which can't keep a generated Secret stable), create the join Secret yourself and set `cluster.bootstrapSecret.existingSecret`. A changed secret applies without restarting the controller. - `daemonSet` (host-network resolvers on every node) comes later. - **Don't raise the replica count of a workload that shares one volume.** Every copy would share one node identity, `state.db`, and query log. Each TelltaleDNS process locks its data directory (`/telltale.lock`) for as long as it runs. A second process pointed at the same directory waits up to 10 s for it, then exits with an error naming the holder (pid, host or pod, and since when). The lock is released when the process exits, even after a crash. Use `mode: scaled` for more pods. With your own manifests, give a single-volume Deployment `strategy: Recreate`, or the new pod can't start until the old one stops. ### Scaling in Kubernetes - **`allInOne` or `scaled`.** `allInOne` is one pod with one volume. It's simple, and it's enough for a home network: one pod answers tens of thousands of queries per second. `scaled` adds resolver pods for availability: a pod can restart or move while the others keep answering. The controller keeps the volume (users, the query log, lists) and the configuration. Resolver pods hold no state, so add or remove them freely (`resolvers.replicas`). - **Don't raise `replicas` on a workload with one volume.** The pods would share one node identity, `state.db`, and query log. The data-directory lock stops the second pod (see above), so use `mode: scaled` instead. - **Each pod has its own cache and rate limits.** A new pod starts with an empty cache, so its hit rate is low for its first minutes. The Cluster page shows each pod's hit rate. Per-client rate limits apply per pod, so a client spread over three pods can send up to three times the limit. - **Spread the pods.** With `externalTrafficPolicy: Local` (needed to see real client addresses), each Kubernetes node sends queries only to its own pods. Use `resolvers.affinity` (for example podAntiAffinity on `app.kubernetes.io/component: resolver`) to put them on different nodes. The Cluster page's **Load balance** check flags a pod that takes more than twice its fair share of its site's queries. - **What one Kubernetes node protects against:** a pod crash, an upgrade, or a pod being rescheduled. The other pods keep answering. It doesn't protect against the node itself going down. For that, run pods on several nodes, or add a node outside Kubernetes (a Pi) as a replica that clients use as their second DNS server. - **On the Cluster page**, the topology shows each site with its nodes and its pods grouped by Kubernetes node. Each line's thickness is that node's share of the queries; amber means not ready or behind on configuration, and red means down. Select a pod to jump to its row (pod name, Kubernetes node, share of queries, cache, and restarts) and its machine. The events count pods joining, pods gone, and restarts in the last hour. - **One IP for DNS, the UI, and the cluster port.** Don't give the `api` or cluster Service the DNS Service's IP (Cilium's `lb-ipam-sharing-key`, MetalLB's `allow-shared-ip`). With `externalTrafficPolicy: Local`, both only share an IP between Services that select the same pods. In `mode: scaled` the DNS Service also selects the resolver pods, so it loses the IP and DNS stops on that address. Instead set `service.dns.alsoServe.api: true` (and `.cluster: true` for a node outside Kubernetes). The DNS Service then carries those ports too, but only the controller listens on them. Keep `service.api` on `ClusterIP` (it still serves the ingress). The install notes warn when an `api` or cluster Service is a LoadBalancer in scaled mode. - **With pods on more than one node, take care.** With `externalTrafficPolicy: Local`, the load balancer announces the IP from a node that has a ready pod of that Service, and that node delivers only to its own pods. If it's a node that runs only a resolver pod, DNS on port 53 keeps working but the UI and cluster ports go nowhere, and which node announces can change after any restart. A node outside Kubernetes that joined through that address then can't reach the controller. - **Safe setups:** a single-node cluster; the controller on every node that can announce the address (`affinity`); a LoadBalancer Service with its own address for the port (`cluster.service.type: LoadBalancer` for the cluster port, with the nodes' `--url` pointing at it, and `service.api.type: LoadBalancer` or an ingress for the UI); or no resolver pods (`resolvers.replicas: 0`). The install notes warn when `alsoServe` is combined with `externalTrafficPolicy: Local` and resolver pods. - **Pods that shut down leave the cluster.** On SIGTERM (scaled in, replaced by a rollout, or deleted) a resolver pod tells the controller it's leaving, and it's dropped from the Cluster page at once (event **Left**). A pod that dies without shutting down (a crash, an out-of-memory kill, a lost node) can't say so: it shows as down and expires after `resolvers.ephemeralTtlSeconds`, as before. Only resolver pods leave this way; the controller and nodes outside Kubernetes stay members across restarts. - **Rollouts without lost lookups.** On shutdown a pod reports not ready, then keeps answering for `drainDelaySeconds` (default 5; passed as `TELLTALE_DRAIN_DELAY_SECS`, which is `[node] drain_delay_secs`) while the Service stops sending it queries, and only then stops. Outside Kubernetes the default is 0. - **Metrics.** With `serviceMonitor.enabled`, Prometheus scrapes the controller and every resolver pod (a headless `-resolver-metrics` Service), so query, SERVFAIL, and latency metrics cover all the traffic, and so do the alerts and SLO rules of `prometheusRule.enabled` (both jobs, `-metrics` and `-resolver-metrics`; before 0.5.0 they watched only the controller). With the kube-prometheus-stack chart, give the rules its `release` label so Prometheus loads them (`prometheusRule.labels: {release: }`). On the controller, per member: `telltale_cluster_peer_info{kube_node, pod}`, `telltale_cluster_peer_queries_per_second`, `telltale_cluster_peer_cache_hit_ratio`, and `telltale_cluster_peer_restarts`, per member. ### Kubernetes workloads as devices Your cluster's pods can be devices like any laptop or TV: named by the workload that runs them, grouped, filtered, and watched. TelltaleDNS reads each pod's owner and calls the pod's address by its **workload**: `media/deploy/jellyfin` for the Deployment `jellyfin` in the namespace `media`, and likewise `sts`, `ds`, `job`, `cronjob` (every run of a CronJob is one device), and `pod` for a bare pod. A Deployment's three replicas are one device, before and after every rollout, so its top domains, anomaly baseline, rhythm, and timeline carry on when its pods are replaced. A new pod often asks DNS before Kubernetes reports its address. The query log still names those first queries by the workload. The dashboard and the anomaly engine wait up to 2 seconds for an address they don't know yet, so a CronJob's short runs count under the CronJob. While this is on, those numbers run up to 2 seconds behind for devices outside the cluster. A restart of TelltaleDNS keeps all of it: each node saves its workload counts hourly and at shutdown (`workload-counts.json` in the data directory), and the restored top lists name workloads, not the pods' addresses. Turn it on in the chart's values: ```yaml kubernetes: enabled: true clusterDns: 10.43.0.10 # kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}' podCidr: 10.42.0.0/16 # optional: PTR lookups for pod and Service addresses go to CoreDNS too serviceCidr: 10.43.0.0/16 # (leave both out if the nodes' own DNS includes TelltaleDNS, below) service: dns: clusterIP: 10.43.0.53 # a fixed address inside the Service network, for the kubelet setting ``` The controller (or the `allInOne` pod) then gets a service-account token and a ClusterRole that can only `get`, `list`, and `watch` pods and namespaces. Resolver pods get no token. The controller sends the workload table to every node in the cluster (a Pi included, so it names the rows it ships), and the nodes ask for it again every minute. The chart also sends `cluster.local` (and the reverse zones of `podCidr` and `serviceCidr`) on to CoreDNS, so Service names keep working. That route is `node_only` (CLU-006): a Pi in the same cluster doesn't get it, since it can't reach CoreDNS, and answers `cluster.local` like any other name. Set Kubernetes options through these values, not as a `[kubernetes]` table in `config`: the chart writes that table. **Pods must ask TelltaleDNS directly.** If they ask CoreDNS and CoreDNS forwards to TelltaleDNS, every query comes from CoreDNS's address. Two ways to do it: 1. **The whole cluster (recommended).** Point the kubelet's cluster DNS at TelltaleDNS's DNS Service first and CoreDNS second. On k3s, in `/etc/rancher/k3s/config.yaml` on every node, then restart k3s: ```yaml kubelet-arg: - "cluster-dns=10.43.0.53,10.43.0.10" # service.dns.clusterIP, then CoreDNS ``` Use `kubelet-arg`, not k3s's own `cluster-dns` option, which also moves CoreDNS's Service (k3s keeps its own value in the kubelet's configuration file, and the flag wins). Restarting k3s leaves running pods running; new pods use the setting, so restart workloads (`kubectl -n media rollout restart deploy`) to move the ones that are running. On kubeadm clusters, set `clusterDNS` in the kubelet's configuration. TelltaleDNS keeps answering the cluster's names through CoreDNS, so Services resolve as before. CoreDNS second keeps lookups working if the DNS Service has no ready pods; images based on Alpine (musl) ask both servers at once, so some of their lookups arrive through CoreDNS and count under it. To start new clusters this way, have the install script write the file before installing k3s (`sudo install -D -m 0600 k3s-config.yaml /etc/rancher/k3s/config.yaml`). TelltaleDNS's own pods keep asking CoreDNS (`kubernetes.ownPodsUseClusterDns`, on by default), so they never depend on their own Service. 2. **One workload at a time** (a good first step before the whole cluster). Give its pods their own DNS settings: ```yaml spec: dnsPolicy: "None" dnsConfig: nameservers: ["10.43.0.53"] searches: ["media.svc.cluster.local", "svc.cluster.local", "cluster.local"] options: [{ name: ndots, value: "5" }] ``` **The nodes' own DNS.** CoreDNS sends everything outside the cluster to the node's DNS servers (its `forward . /etc/resolv.conf`), so give each node two, for example a Pi and TelltaleDNS's LoadBalancer address. On Ubuntu with NetworkManager, add the second to the netplan file's `nameservers.addresses`, apply it without taking the link down (`sudo netplan generate && sudo nmcli connection reload && sudo nmcli device reapply `), then `kubectl -n kube-system rollout restart deploy/coredns`. Once the nodes' DNS includes TelltaleDNS, leave `kubernetes.podCidr` and `serviceCidr` out: CoreDNS passes reverse names it doesn't know (`fallthrough in-addr.arpa`) to the node's servers, so an unknown pod address's PTR would loop between TelltaleDNS and CoreDNS. Traffic from a pod to a Service address keeps the pod's own address (kube-proxy and Cilium don't rewrite it), so "Client IPs appear masked" isn't affected. With `ndots: 5` you'll see questions like `grafana.com.media.svc.cluster.local`. They go to CoreDNS and come back NXDOMAIN: that's how Kubernetes searches, and the traffic map shows it as it is. **Groups.** A workload's group is the first of: 1. the label `telltaledns.io/group=` on its pods; 2. the same label on its namespace (`kubectl label namespace media telltaledns.io/group=media`); 3. the first matching rule of `kubernetes.map` (`[{namespace: "media*", group: "iot"}]`, with optional `kind` and `name`), or one made in the UI; 4. `kubernetes.defaultGroup`; 5. otherwise its network's group, or `default`, like any device you haven't named. A label naming a group that doesn't exist is logged once and ignored. Labels take effect within seconds, on every node, with no configuration change. Everything a group does then applies to the namespace: lists, quick rules, blocked services, schedules, upstreams. A `$client` rule names a workload by its name (`||ads.example^$client=media/deploy/jellyfin`). **In the UI**, the Clients page gets a **Workloads** tab: each workload with its kind, namespace, group and where it came from, its pods, its queries and blocks in the last 24 hours, and when it last asked. Click one for its day on the timeline. Everywhere a device shows, a workload has a small cube and its name, with the pod and address on hover. "Name this device…" is off for workloads, because Kubernetes names them. "Add to group…" previews and then saves a rule for exactly that workload. A label is the Git way to do the same. Explain says which workload and pod asked and where the group came from. The query log keeps each pod's own address. Each row is named by the workload that held that address when the query was made, so an address reused after a rollout doesn't relabel the past. Filter it by workload with `?client=media/deploy/jellyfin`. **API and agents.** `GET /api/v1/kubernetes/workloads` lists the workloads (filter by `namespace` or `group`), and `GET /api/v1/kubernetes/status` says whether this node watches or takes the table from the cluster. `PUT /api/v1/kubernetes/map/{namespace}/{kind}/{name}` with `{"group": "iot"}` writes a rule (`*` for any kind or name). The MCP tools are `list_workloads` and `get_workload`. `plan_assign_client` with a workload's name plans a rule. A workload's name works as `client` in the query log, the timeline, rhythm, identity, and explain. **When the API server can't be reached**, DNS answers as always. Pods keep the names they had, and new pods are plain addresses until the watch is back. After 15 minutes the health level turns **degraded** (`kubernetes_stale`), and the `TelltaleDNSKubernetesWatchStale` alert fires. Metrics: `telltale_kubernetes_pods`, `telltale_kubernetes_workloads`, `telltale_kubernetes_table_age_seconds`, `telltale_kubernetes_watch_errors_total`, `telltale_kubernetes_sync_total{result}`, and `telltale_kubernetes_watching`. Limits: one Kubernetes API per cluster; pods on the host network are the node's address and stay that; a deleted pod's address stays its own for 30 seconds (queries still in flight), then the history keeps who held it for 30 days; at most 20,000 pod addresses (the metric `telltale_kubernetes_addresses_dropped` says when more are left out). TelltaleDNS doesn't answer `cluster.local` itself: CoreDNS does. ## Seeing real client IPs Per-device statistics, groups, and rules need each query's real sender. TelltaleDNS checks this continuously: when more than 90% of the last 10 minutes' queries (at least 100) came from 3 or fewer *infrastructure* addresses, the UI shows a "Client IPs appear masked" banner, `GET /api/v1/system/info` includes `clientIpsMasked` with the evidence, the metric `telltale_client_ips_masked` is 1, and the log says so once. Infrastructure addresses are loopback, this host's default gateways (a Docker bridge or a Kubernetes pod's gateway; on the LAN, a router), the node addresses the Helm chart passes in `TELLTALE_NODE_IPS`, and any networks in `[clients] infrastructure`: ```toml [clients] infrastructure = ["10.42.0.0/16"] # e.g. your cluster's pod network ``` Common causes and fixes: - **Kubernetes, `externalTrafficPolicy: Cluster`**: the node rewrites the sender. Use `Local` (the chart's default) or `hostNetwork: true`. - **Docker with a userland proxy** (IPv6 or `127.0.0.1` port publishing): run with `network_mode: host`, or publish on the LAN address. - **A router that forwards DNS** (its own resolver points at TelltaleDNS and clients use the router): hand out TelltaleDNS's address in DHCP instead. If you can't, enable the router's EDNS MAC option (dnsmasq `add-mac`) and list the router in `[clients] trust_edns_mac_from` so devices are recognized by MAC. ## Upstream presets Instead of looking up addresses, start from a preset: ```sh telltale presets list # Cloudflare, Google, Quad9, AdGuard, Mullvad, Control D, NextDNS, ... telltale presets show quad9 --proto tls,https --group default >> telltale.toml telltale presets show nextdns --param profile=abc123 # templated presets need your account ID ``` `show` prints explicit `[[upstream]]` entries plus a `fastest` group. Your config always lists exactly what's used and never depends on the catalog, which only helps you write it. The catalog covers every Pi-hole preset plus the common encrypted resolvers; a nightly job checks that every entry still answers. That includes the DoQ (`quic://`) endpoints. ## Encrypted DNS for your devices (DoT, DoH, and DoQ) Phones, laptops, and browsers can reach TelltaleDNS over DNS over TLS (Android's "Private DNS", RFC 7858), DNS over HTTPS (browsers, iOS and macOS profiles, RFC 8484), or DNS over QUIC (RFC 9250; AdGuard and other apps, some routers), so nobody on the network path can read or change their lookups: ```toml [[listen]] proto = "dot" addr = "0.0.0.0:853" tls = { cert = "/etc/telltale/tls.crt", key = "/etc/telltale/tls.key" } [[listen]] proto = "doh" addr = "0.0.0.0:443" path = "/dns-query" # default; /dns-query/ also works tls = { cert = "/etc/telltale/tls.crt", key = "/etc/telltale/tls.key" } [[listen]] proto = "doq" addr = "0.0.0.0:853" # UDP 853 (DoT uses TCP 853: both fit) tls = { cert = "/etc/telltale/tls.crt", key = "/etc/telltale/tls.key" } [[listen]] proto = "doh3" # DoH over HTTP/3: UDP 443 next to the TCP one addr = "0.0.0.0:443" tls = { cert = "/etc/telltale/tls.crt", key = "/etc/telltale/tls.key" } ``` - **Certificate:** a PEM chain (leaf first) and key, valid for the name devices use, e.g. `dns.example.com` from Let's Encrypt (DNS-01 works for internal names). Both files are re-read within 10 seconds of changing, so renewals (certbot, cert-manager) need no restart; a broken renewal keeps the old certificate and logs a warning. - **DoH** speaks HTTP/2 and HTTP/1.1, `GET ?dns=` and `POST application/dns-message`, and answers with `Cache-Control: max-age` set to the answer's smallest TTL. DoT uses ALPN `dot` with RFC 7766 pipelining, like plain TCP. With a `doh3` listener, DoH also speaks HTTP/3 (ALPN `h3`, the same paths and client IDs), and HTTP/2 answers carry `Alt-Svc: h3=":443"` so browsers and apps that support it switch. DoQ uses ALPN `doq`: each query has its own QUIC stream, so a slow answer never holds up the others, and an idle connection closes after 30 s. 0-RTT is off (an early query could be replayed). - **Devices identify themselves:** with a wildcard certificate (`dns.example.com` and `*.dns.example.com`), a device configured with `kids-tablet.dns.example.com` (Android Private DNS, DoT, DoH, or DoQ) or the URL `https://dns.example.com/dns-query/kids-tablet` gets the client ID `kids-tablet`, which `[[client]] match = ["id:kids-tablet"]` recognizes wherever the device is, even on mobile data. The path wins over the name. - **Behind a load balancer:** `proxy_protocol = true` on a `tcp`, `dot`, or `doh` listener makes it read the client's address from a PROXY protocol v2 header (HAProxy, Traefik, AWS NLB, ...). Every connection must then start with one, so only enable it when the balancer sends it, and don't let clients reach the listener directly. `LOCAL` connections (the balancer's health checks) keep the socket address. - **Slow and silent clients:** on `tcp` and `dot` listeners a connection that sends nothing for the idle timeout (10 s) is closed, and so is one whose client stops reading answers, so a stalled client can't hold one of the 1,024 connection slots. A query longer than 4,096 bytes (the UDP receive buffer's size; real queries are a few dozen bytes) closes the connection. Closures are counted in `telltale_tcp_idle_closed_total` and `telltale_tcp_stalled_closed_total`. - **Connection limits, per listener** (`tcp`, `dot`, `doh`, `doh3`, `doq`): `max_connections` (default 1,024) is how many connections it holds open at once, and `max_connections_per_address` (default 32, `0` = no limit) is how many one client address may hold, so a single host can't take every slot. An IPv6 client counts per /64, and with `proxy_protocol = true` the client's address is the one in the PROXY header. Connections over a limit are refused (counted in `telltale_tcp_rejected_per_address_total`, `telltale_doh_rejected_per_address_total`, `telltale_doq_rejected_per_address_total`, next to the `*_rejected_total` counters for the overall limit). **Behind a load balancer that hides client addresses** (no PROXY protocol, source NAT, or Kubernetes `externalTrafficPolicy: Cluster`), every client looks like the balancer, so raise `max_connections_per_address` or set it to `0`. Changing a limit restarts that listener (open connections finish first). ```toml [[listen]] proto = "dot" addr = "0.0.0.0:853" tls = { cert = "/etc/telltale/tls.crt", key = "/etc/telltale/tls.key" } max_connections = 2048 max_connections_per_address = 64 ``` - Metrics: queries are counted per transport (`telltale_queries_total{proto="dot"|"doh"|"doq"}`), plus `telltale_doq_connections_total`, `telltale_doq_rejected_total` and `telltale_doh_rejected_total` (connections refused at the 1,024 cap, which `doq` and `doh3` enforce like `tcp`, `dot`, and `doh`), `telltale_doq_protocol_errors_total`, `telltale_doh_requests_total`, `telltale_doh_bad_requests_total`, `telltale_tls_handshake_failures_total`, and `telltale_proxy_protocol_rejected_total`. - In Kubernetes, enable `encrypted.dot` / `encrypted.doh` (plus `encrypted.doh.http3`) / `encrypted.doq` in the chart with a certificate from an existing Secret or cert-manager (`encrypted.tls.certManager`); the ports join the DNS LoadBalancer, so client addresses survive (`externalTrafficPolicy: Local`). ## Recursive resolution (no forwarder) TelltaleDNS can resolve names itself, from the root servers down, like `unbound` next to a Pi-hole, so no public resolver sees your lookups: ```toml [[upstream]] name = "recursive" url = "recursive://" [[upstream_group]] name = "default" members = ["recursive"] # Optional (these are the defaults): # [upstream.recursive] # qname_minimization = true # each server sees only as much of the name as it needs (RFC 9156) # case_randomization = false # 0x20: random letter case, checked in answers (spoofing defense) # ipv6 = false # also ask servers over IPv6 (needs an IPv6 route) # stub_zone / stub_servers # start one zone at its own servers: see Stub zones ``` - **Private by default:** with QNAME minimization the root servers see only `com`, the `com` servers only `example.com`, and only `example.com`'s own servers see `www.example.com`. If a server mishandles the shortened question, the full one is sent (relaxed mode). - **Safe:** a random port and ID per query; records are only accepted from servers responsible for them, and server addresses ("glue") only from the zone that delegates to them. With `case_randomization` (off by default), each query asks for a random mix of upper and lower case, and only an answer that echoes it exactly is accepted: it adds about one bit of entropy per letter to the 32 bits of ID and port an off-path spoofer must guess, so it is worth turning on for a recursive resolver. An answer with the case changed is ignored like any other stray packet while the real one is awaited; only a server that never echoes the case is remembered (after one timeout) and asked without it. - **Fast after the first time:** zone cuts and server addresses are remembered (bounded, by their TTLs), answers go to the normal cache, and servers are picked by measured response time. A first lookup in a new zone takes a few round trips (about 30 to 400 ms); after that, one. - **Hedged:** when a server is slower than usual (1.5 times its usual response time, at least 150 ms), the next server is asked too, and the first good answer wins, so one sluggish server doesn't hold up a lookup. - **Root priming** (RFC 8109): on first use, the built-in root server addresses are only used to ask for the root's current list, which is then remembered for a day. A root server that changes address doesn't need a new TelltaleDNS. - **DNSSEC:** with `[dnssec] mode = "validate"` the signatures come along and are checked from the root's trust anchor, as for forwarded answers. - It's an upstream like any other: mix it into a group with forwarders (for example `strategy = "failover"` with a public resolver as the fallback), or route only some domains through it. - Limits that stop loops and broken zones: 16 CNAME hops, 32 referrals, 96 queries per question; servers get at most 1.2 s each. - Non-existent names that validated NSEC records already cover are answered without asking (RFC 8198; see DNSSEC below). - Moving from Pi-hole with unbound, or from Technitium without forwarders: the importers note it (Technitium's import uses `recursive://` directly). ## Encrypted upstreams ```toml [[upstream]] name = "cloudflare-dot" url = "tls://1.1.1.1" # DNS over TLS, port 853 tls_server_name = "cloudflare-dns.com" # name on the certificate [[upstream]] name = "quad9-doh" url = "https://dns.quad9.net/dns-query" # DNS over HTTPS (HTTP/2) bootstrap = ["9.9.9.9", "149.112.112.112"] # how to look up dns.quad9.net itself [[upstream]] name = "google-h3" url = "h3://8.8.8.8/dns-query" # DoH over HTTP/3 (or https:// with http_version = "3") tls_server_name = "dns.google" [[upstream]] name = "adguard-doq" url = "quic://94.140.14.14" # DNS over QUIC, port 853 tls_server_name = "dns.adguard-dns.com" ``` - Certificates are verified against the built-in Mozilla root set, so no CA files are needed, even in a minimal container. `tls_insecure_skip_verify = true` turns verification off (a warning is logged; don't use it on untrusted networks). - **Hostname upstreams** are looked up through `bootstrap` servers, or the system resolvers from `/etc/resolv.conf` when `bootstrap` is empty (never through TelltaleDNS itself), and the result is cached for its TTL. To skip the lookup, put the IP in the URL and set `tls_server_name`. - Connections are kept open and reused: many queries share one DoT connection (`pool_size`, default 4; closed after `idle_timeout_ms`, default 30 s), and DoH multiplexes every query over a single HTTP/2 connection. A new connection finishes its handshake even when the query that started it runs out of time, so the next query finds it ready. A connection that stops answering altogether (queries waiting 3 s with nothing coming back, as when something along the way drops it silently) is closed, and its queries retry at once on a new one. The `telltale_upstream_connections_*` metrics show how often connections open, why they close, and whether TLS sessions resume. DoQ keeps one QUIC connection per upstream with a stream per query, reconnects when it closes, and never uses 0-RTT; DoH over HTTP/3 does the same with a request per query. - `http_version = "3"` on an `https://` upstream uses HTTP/3 only; `"auto"` (the default) and `"2"` use HTTP/2. - `doh_method = "get"` sends DoH queries as `GET /dns-query?dns=…` (RFC 8484 §4.1) instead of POST, for providers or HTTP caches that prefer it. The query ID is 0 either way, so identical questions make identical URLs. ### DNSCrypt and DNS stamps Use a DNSCrypt server (as dnscrypt-proxy does next to a Pi-hole), or any server from a DNS stamp list (the public list at dnscrypt.info), by pasting its `sdns://` stamp as the URL: ```toml [[upstream]] name = "adguard-dnscrypt" url = "sdns://AQMAAAAAAAAAETk0LjE0MC4xNC4xNDo1NDQzINErR_JS3PLCu_iZEIbq95zkSV2LFsigxDIuUso_OQhzIjIuZG5zY3J5cHQuZGVmYXVsdC5uczEuYWRndWFyZC5jb20" ``` - **DNSCrypt stamps** become a DNSCrypt v2 upstream: the server's certificate is fetched and checked against the provider key in the stamp, then every query is encrypted (X25519 with XChaCha20-Poly1305 or XSalsa20-Poly1305, whichever the certificate offers) and padded, over UDP with TCP for large answers. Certificates are refreshed hourly and when they expire. - **DoH, DoT, and DoQ stamps** become the matching `https://`, `tls://`, or `quic://` upstream, with the stamp's host name as the TLS name and its address (if any) pinned, so no bootstrap lookup is needed. - **Certificate hashes in stamps are enforced:** when a DoH, DoT, or DoQ stamp lists hashes, some certificate in the chain the server presents (usually an intermediate CA) must match one of them, on top of the normal certificate check (and even with `tls_insecure_skip_verify`). If a provider changes CAs before its published stamp is updated, the upstream fails with "no certificate in the server's chain matches the stamp's hashes": take the newer stamp, or use the plain `https://` / `tls://` URL, which checks the certificate the usual way. - Tested against AdGuard, OpenDNS, CleanBrowsing, Comodo, and cryptostorm's DNSCrypt servers. *Not supported:* DNSCrypt over IPv6-only paths without an IPv6 route. **Anonymized DNSCrypt.** Add a relay, and the queries go through it: the relay sees your address but can't read the queries, and the resolver reads the queries but sees only the relay's address. Pick a relay run by someone other than the resolver (the public relay list at dnscrypt.info has stamps): ```toml [[upstream]] name = "adguard-anon" url = "sdns://AQMAAAAAAAAAETk0LjE0MC4xNC4xNDo1NDQzINErR_JS3PLCu_iZEIbq95zkSV2LFsigxDIuUso_OQhzIjIuZG5zY3J5cHQuZGVmYXVsdC5uczEuYWRndWFyZC5jb20" relay = "sdns://gQ8yMDMuMC4xMTMuNzo0NDM" # a relay stamp (this one is 203.0.113.7:443), or "ip:port" ``` - Everything goes through the relay, the certificate fetch included, over UDP and over TCP for large answers. - `relay` is only valid on a DNSCrypt (`sdns://`) upstream. ### Client subnet (ECS) Some CDNs pick a server near the client by the subnet a resolver sends along (EDNS Client Subnet, RFC 7871). TelltaleDNS never sends your devices' subnets. When a far-away public resolver gets you slow CDN answers, you can send a subnet of your choice instead, for example your ISP's public block: ```toml [[upstream]] name = "quad9-ecs" url = "tls://9.9.9.11" # Quad9's ECS-enabled service tls_server_name = "dns11.quad9.net" ecs = "203.0.113.0/24" # sent with every query (default: "strip", nothing sent) ``` - The same subnet goes with every query, so cached answers stay valid for everyone and no device's address leaves your network. - ECS options from your devices are never forwarded, and an upstream's ECS reply never reaches them. When TelltaleDNS serves clients on public addresses (a hosted resolver, or a VPS for a family spread over several ISPs), `ecs = "client"` sends each client's own subnet instead: ```toml [[upstream]] name = "quad9-ecs" url = "tls://9.9.9.11" tls_server_name = "dns11.quad9.net" ecs = "client" # each client's /24 (IPv4) or /56 (IPv6) ``` - Only the subnet goes out: the address cut to /24 or /56, the privacy defaults RFC 7871 recommends. - Clients on private, loopback, link-local, CGNAT (100.64.0.0/10), or ULA (fc00::/7) addresses send nothing, since their subnet would tell a CDN nothing. On a home network that's every device, so `"client"` only matters for public clients. - Answers from an upstream group with a `"client"` member are cached per subnet, so one subnet's answer never reaches another. They're also left out of the cache file kept across restarts. - The subnet comes from the client's address (after PROXY protocol, if enabled). ECS options your clients send are still never forwarded. ### Through a proxy (Tor) Send an upstream's traffic through a SOCKS5 or HTTP proxy, for example Tor, so the resolver never sees your address: ```toml [[upstream]] name = "quad9-tor" url = "tls://dns.quad9.net" # the proxy resolves the name: no bootstrap, no leak proxy = "socks5://127.0.0.1:9050" # or "http://127.0.0.1:3128", with optional user:pass@ ``` - Works with `tcp://`, `tls://`, and `https://` upstreams, and with `udp://` through a SOCKS5 proxy that supports UDP ASSOCIATE (RFC 1928 §7, e.g. Dante). A truncated UDP answer is asked again over TCP through the same proxy. A few associations are kept open and reused. - Tor and HTTP proxies don't carry UDP: use `tcp://` (or `tls://`) with them. - A hostname in the URL is passed to the proxy as a name, so the lookup happens at the proxy's end (with Tor, at the exit). - Credentials in the proxy URL (`socks5://user:pass@host:port`) never appear in logs. ### Pinning and client certificates ```toml [[upstream]] name = "office-dot" url = "tls://10.0.0.53" tls_server_name = "dns.office.example" tls_ca = "/etc/telltale/office-ca.pem" # trust this CA for this upstream only tls_client_cert = "/etc/telltale/client.pem" # servers that require a client certificate (mTLS) tls_client_key = "/etc/telltale/client.key" spki_pins = ["kXbB2F3...base64...="] # the server's key must be one of these ``` - **Pins** are the base64 SHA-256 of the server certificate's public key (SPKI), checked after the usual verification. With `tls_insecure_skip_verify = true` only the pin is checked, which suits a self-signed server you pin. List two pins (current and next key) to rotate keys without an outage. To get a server's pin: ```sh openssl s_client -connect 10.0.0.53:853 -servername dns.office.example /dev/null \ | openssl x509 -pubkey -noout | openssl pkey -pubin -outform der \ | openssl dgst -sha256 -binary | base64 ``` - The files are read when the configuration loads (and on every reload); a missing or unreadable file is a configuration error. ### Plugin upstreams Anything that can answer DNS can be an upstream, in any language, without linking into TelltaleDNS: a plugin speaks plain DNS messages over a stream (a 2-byte length, then the message, as DNS over TCP does) on a Unix socket. ```toml [[upstream]] name = "magicdns-bridge" url = "unix:///run/magicdns.sock" # a socket another service listens on [[upstream]] name = "consul" url = "exec:///usr/local/bin/consul-dns-plugin" args = ["--datacenter", "home"] ``` - With `exec://`, TelltaleDNS starts the program and tells it where to listen in `TELLTALE_PLUGIN_SOCKET` (under `/plugins/`; `TELLTALE_UPSTREAM` has the upstream's name). If it exits, it's restarted after 1 s, then 2, 4, ... up to 60 s (back to 1 s once it has run for a minute). Its output goes to TelltaleDNS's log. It's stopped when the configuration changes or TelltaleDNS stops. - A plugin can't hurt the resolver: it's another process, every exchange has the upstream's timeout, and a dead plugin is a failing upstream like any other (health, breaker, the group's other members). ## How a query is answered ```mermaid flowchart LR Q[query] --> P{parse} P -- malformed --> E[FORMERR / NOTIMP / BADVERS] P --> C{cache} C -- fresh hit --> A[answer, TTLs counted down] C -- miss / expired --> U[upstream group] U -- answer --> S[cache + answer] U -- slow or failing, have stale data --> ST[stale answer + EDE 3] U -- all failed --> SF[SERVFAIL + EDE 22] ``` - **Cache:** answers are cached for their TTL (clamped by `[cache] min_ttl`/`max_ttl`). Negative answers are cached when the upstream includes an SOA. SERVFAIL is cached for 5 seconds, and never served stale afterwards: an expired failure is asked again. Hot entries are refreshed in the background just before they expire. - **Upstreams:** if an upstream is slow, the next one is tried *in parallel* rather than after a timeout, and the first good answer wins. An upstream that fails 3 times in a row (or more than half the time) is benched for 10 seconds, doubling up to 5 minutes, then probed again. A SERVFAIL or REFUSED answer counts toward "more than half the time" but not toward "3 in a row", so a client retrying one broken domain can't bench a healthy upstream. Identical concurrent questions share one upstream request. - **Serve-stale:** if upstreams don't answer within 1.8 s (`[cache] stale_answer_client_timeout_ms`) and an expired answer is still in the cache (up to a day old by default), it's served with TTL 30 and Extended DNS Error 3 ("Stale Answer"), while the refresh continues in the background. - **Answering recently expired names at once:** devices keep answers themselves, so most repeat questions reach the resolver just after the answer expired, and a plain cache misses them. With `stale_answer_client_timeout_ms = 0` an expired answer is given at once (TTL 30, EDE 3) while it's refreshed in the background, the way Pi-hole's cache optimizer works by default. `stale_max_age` bounds how old it may be (an hour matches Pi-hole), and `min_ttl` keeps short-lived answers a little longer (Technitium's default is 10 s): ```toml [cache] min_ttl = 10 stale_answer_client_timeout_ms = 0 # answer an expired name at once, refresh it in the background stale_max_age = 3600 ``` These answers have the status `refreshed`, so `stale` keeps meaning "upstreams failed or were slow" (or the resolver was too busy to start the refresh). They cost about as much as a fresh cache hit. The trade-off is that a name whose address changed within the hour is answered once with the old address (for 30 s) while the new one is fetched. `[cache]` is per node: set it on every node of a cluster. - **What the dashboard counts:** the **Cache hits** tile is every answer given from the cache: fresh, `refreshed`, or `stale`, out of the answers that weren't blocked or local. In the Advanced view its second line splits it ("41% fresh · 22% refreshed") and the status chart shows `refreshed` on its own. The Cache page's "Answered from the cache" is the same number per node, with the fresh share beside it. Prometheus counts a refreshed or stale answer as a cache miss plus one `telltale_cache_stale_served_total`; the Grafana dashboard's **Answered from cache** panel shows both lines. - **Privacy:** queries to upstreams carry a fresh random ID and source port, and none of the client's EDNS options (no client subnet, cookies, or MAC addresses are forwarded). - **Loop protection:** outbound queries carry a random per-process tag. If one comes back to us (an upstream that forwards to TelltaleDNS), it's dropped and an error is logged instead of looping forever. ## Local records Names on your network, answered by TelltaleDNS itself (authoritatively, before cache and upstreams): ```toml [[record]] name = "nas.home.arpa" type = "A" # A, AAAA, CNAME, PTR, TXT, MX, SRV value = "192.168.1.10" [[record]] name = "*.dev.home.arpa" # wildcard: every name below dev.home.arpa type = "A" value = "192.168.1.20" [[record]] name = "_sip._udp.home.arpa" type = "SRV" value = "0 5 5060 pbx.home.arpa" # priority weight port target (MX: "preference exchange") [local] hosts_files = ["/etc/telltale/hosts"] # "IP name [alias...]" lines auto_ptr = true # reverse lookups for every A/AAAA (default) default_ttl = 300 ``` - A name with no record of the requested type gets an empty authoritative answer (NODATA). Names that aren't local go to the upstreams as usual. - CNAMEs are followed within local data; if the target is elsewhere, the client follows it. - Hosts files skip `0.0.0.0`, `::`, and loopback entries (those are blocklists or this machine, not network hosts). - `telltale config check` validates record values and reads the hosts files. - **In the web UI:** **Names on my network** lists every local name grouped by domain. Operators can add, edit, and remove names (A, AAAA, and CNAME in Simple; every type in Advanced), run **Set up my home domain** (home.arpa by default) for a first device, and **Send a domain to another server** (conditional forwarding). Each change is previewed in a sentence and a diagram before it's saved, and every name has a **Test** link to Explain. Names from the config files are shown read-only. Through the API: `GET /api/v1/records`, `PUT`/`DELETE /api/v1/records/{name}` with `{"records": [{"type": "A", "value": "192.168.1.10"}]}`, and `GET /api/v1/forwards`, `PUT`/`DELETE /api/v1/forwards/{domain}` with `{"servers": ["10.0.0.53"]}` (add `"stub": true` for a [stub zone](#stub-zones-the-zones-own-servers)); all take `?dryRun=true`, `If-Match`, and `Idempotency-Key` like devices. - **Moving from another DNS server:** `telltale import zone exported.zone > records.toml` converts a zone file (Technitium's Zones → Export Zone, BIND, PowerDNS) into `[[record]]` entries for your config. The header lists anything with no local equivalent (SOA and NS aren't needed: TelltaleDNS answers these names itself). Names under the zone that you don't import still resolve normally, so a split-horizon domain keeps working: internal names answer locally, everything else publicly. ### Local zones A zone makes TelltaleDNS the authority for a whole domain: names in it are answered from it, and names it doesn't have get NXDOMAIN (instead of going to the upstreams, as names not in `[[record]]` do). ```toml [[zone]] name = "lab.example.com" file = "/etc/telltale/lab.example.com.zone" # an RFC 1035 zone file (BIND, Technitium, PowerDNS) [[zone]] name = "corp.example.com" groups = ["office"] # split horizon: only these groups see this zone negative_ttl = 300 [[zone.record]] name = "intranet.corp.example.com" type = "A" value = "10.10.0.5" ``` - Answers are authoritative (AA). The zone's apex and names that only have names below them exist (no data, not NXDOMAIN). Negative answers carry the zone's SOA, owned by its apex, with `negative_ttl`. - **SOA and NS at the apex** (`dig SOA home.example`, `dig NS home.example`) answer with the zone file's own records. Without a file, or a file without them, they're made up as resolvers serving local zones do: `localhost.` as the name server, `hostmaster.` as the contact, and `negative_ttl` as the SOA minimum. - **Split horizon:** with `groups`, only those groups' devices see the zone; everyone else resolves the domain normally (the public answer). Several zones for the same domain can serve different groups. - The most specific zone wins (`a.corp.example.com` over `corp.example.com`). Zones are answered before filtering, like local records. - Zone files use the record types local records support (A, AAAA, CNAME, PTR, TXT, MX, SRV), plus the apex's SOA and NS. Changes need a reload (`SIGHUP`). ## Filter lists > **Status:** lists are downloaded, compiled, and **enforced** per client group, including names reached through a CNAME. ```toml [[list]] name = "hagezi-pro" # ID: lowercase letters, digits, - and _ url = "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/pro.txt" [[list]] name = "family-allow" kind = "allow" # block (default) or allow match = "exact" # subtree (default: the name and everything below it) or exact path = "/etc/telltale/allow.txt" # a local file, re-read on every refresh [[list]] name = "manual" rules = ["||ads.example.com^", "@@||cdn.example.com^"] [filter] refresh_secs = 86400 # every 24 h (per-list `refresh_secs` overrides; minimum 900) fetch_concurrency = 4 fetch_timeout_secs = 120 # per attempt, including the download fetch_retries = 3 # network errors, HTTP 5xx, and 429 are retried with backoff max_list_bytes = "64MiB" # per-list `max_bytes` overrides max_invalid_percent = 50 # refuse a download if more than this % of its rule lines are invalid max_shrink_percent = 50 # hold a download that lost more than this % of the list's lines (100 = off) compile_threads = 0 # 0 = auto (first compile: half the cores, 1-4; recompiles: 1) compile_memory = "128MiB" # sort budget before spilling to disk fsync = true # sync a compiled snapshot to disk before publishing it max_regexes = 1000 # regex rules kept per snapshot (0 = no limit); the rest are reported allow_link_local_urls = false # true: allow list URLs on 169.254.0.0/16 or fe80::/10 ``` - **Link-local and cloud-metadata addresses are refused** (169.254.0.0/16, fe80::/10, AWS's fd00:ec2::254), whatever name or redirect leads there, so a list URL can't read what a cloud instance serves there. Private LAN addresses are fine. When an agent's pre-save check can't reach a list or an upstream, it's told only that it was unreachable; the UI and the audit log show why. - **Downloads are polite:** after the first download, a refresh sends `If-None-Match`/`If-Modified-Since`, so an unchanged list costs one small request. Redirects are followed, except from `https` to `http`. - **A failed refresh never loses a list.** The last good copy stays in use. A failing list is retried after 5 minutes, then 10, 20, and so on up to hourly, instead of waiting a whole day. Responses that can't be a list are rejected: an empty body, an HTML page from a captive portal, a compressed body (`Content-Encoding` other than `identity`), or text that is mostly not rules (a JSON error from a CDN, a mirror that changed format). "Mostly" is `[filter] max_invalid_percent` (default 50: more invalid lines than rules); cosmetic and unsupported rules don't count as invalid, and `100` turns the check off. - **A list that suddenly shrinks waits for a second look.** When a URL list's new download has lost more than `[filter] max_shrink_percent` of the stored copy's lines (default 50: a list of 2 million names arriving with fewer than 1 million), it isn't used yet: a truncated mirror or a source caught mid-rewrite would otherwise drop half your blocking. The previous copy keeps answering, the Lists page says why, and the next scheduled refresh accepts the smaller list if the source serves the same content again. **Refresh now** accepts it at once. Lists under 1,000 lines, local files, and inline rules are never held; `100` turns the check off. - Lists are stored compressed in `/lists/` (`.src.zst` plus `.meta.json`). Removing a list from the config deletes its files; `enabled = false` keeps them. - List hostnames are looked up through the system resolvers (minus TelltaleDNS's own listeners). If the system resolver *is* TelltaleDNS, the lookup goes through it, which is safe because DNS is answering before downloads start. - Downloads run in the background. DNS starts and answers without waiting for them, and a failing download never affects answers. - Adding, removing, or editing lists applies on reload (`SIGHUP`). `[filter]` changes need a restart. - To refresh every list now instead of waiting for `refresh_secs` (like `pihole -g`), send `SIGUSR1` (`kill -USR1 `, or `docker kill -s USR1 telltale`). Unchanged lists cost one conditional request each, and the filter is recompiled only if something changed. Fetch now, outside the server (same config and data directory): ```sh telltale lists fetch -c telltale.toml # list result bytes lines note # hagezi-pro updated 5049934 227434 # manual unchanged 39 2 ``` ### List syntax Every common format works, and formats can be mixed in one list: | Line | Meaning | |---|---| | `0.0.0.0 ads.example.com` (any IP, several names allowed) | hosts file: block `ads.example.com` (`localhost` and similar lines are skipped) | | `ads.example.com` | the name and everything below it (or only the name, with `match = "exact"`) | | `*.example.com` or `.example.com` | everything below `example.com`, but not `example.com` itself | | `\|\|ads.example.com^` | AdBlock: the name and everything below it | | `\|ads.example.com^` | AdBlock: only the name | | `@@\|\|cdn.example.com^` | exception: allow, even in a blocklist | | `\|\|ads.example.com^$important` | wins over ordinary allow rules | | `\|\|example.com^$dnstype=AAAA\|~A` | only for these query types (`~` excludes) | | `\|\|example.com^$client=192.168.1.0/24\|'Kids tablet'` | only for these clients (`~` excludes) | | `\|\|example.com^$denyallow=mail.example.com` | block `example.com` except these names below it | | `\|\|ads.example.com^$badfilter` | cancels the same rule without `$badfilter` | | `\|\|nas.example^$dnsrewrite=192.168.1.20` | answer with this address instead (also an IPv6 address, a name for a CNAME, `NXDOMAIN`/`REFUSED`, or the full `NOERROR;A;…`, `NOERROR;AAAA;…`, `NOERROR;CNAME;…`, `NXDOMAIN;;` forms) | | `@@\|\|nas.example^$dnsrewrite` | turns rewrites off for these names | | `/^ad[0-9]+\./` | AdBlock regex | | `(^\|\.)doubleclick\.net$` | Pi-hole regex; add `;querytype=A,AAAA` (or `=!A` to exclude) or `;invert` | | `# ...`, `! ...`, `[Adblock Plus 2.0]` | comments | **Rewrites in lists** (`$dnsrewrite`, AdGuard's syntax): a matching rewrite wins over blocking, and `$client`, `$dnstype`, and `$denyallow` narrow it as for other rules. Several address rewrites for one name answer together. A name rewritten only to IPv4 answers an IPv6 question with an empty NOERROR (and the other way round). A CNAME rewrite is resolved like any name. The query log shows the list. Rewrites of other record types (MX, TXT, HTTPS, …) are counted as unsupported. Rewrites in the configuration (`[[group.rewrite]]`) are per group, and come after list blocking. Names are lowercased, internationalized names are converted to punycode, and a trailing dot is ignored. Regexes match the lowercase query name and can't use backreferences or lookaround: the regex engine runs in linear time, so a pattern can't stall a query. Rules that only make sense in a browser are counted as *unsupported* and skipped, not treated as errors: cosmetic rules (`##`), URL paths (`||example.com/ads.js`), wildcards inside names (`ads*.example.com`), IP rules (`||192.0.2.1^`, which filter answers rather than names), and modifiers such as `$third-party` or `$ctag`. A rule with a modifier TelltaleDNS doesn't support is skipped entirely. Applying it without the modifier would block more than the list author intended. See what a list contains and which lines were skipped: ```sh telltale lists check -c telltale.toml # adguard-dns: 179561 lines, 177414 rules, 1658 comments, 0 ignored, 489 unsupported, 0 invalid # L79 unsupported: IP address rule (response IP filtering): ||194.63.143.96^ telltale lists check -c telltale.toml --list manual --rules # every rule in canonical form ``` It exits non-zero if any list has invalid lines. ### Compiling Whenever a list's content changes, a list is added or removed, or a list's `kind`/`match` changes, TelltaleDNS compiles every enabled list into a new **filter snapshot** in `/snapshots//`. The three newest snapshots are kept. A restart with unchanged lists reuses the newest snapshot instead of compiling again, and a failed compile keeps the previous one. - Compiling runs in the background at the lowest CPU priority (`SCHED_IDLE` on Linux), and never pauses or locks query handling. The new snapshot replaces the old one atomically: queries in flight finish with the old one, and the old one is freed on a background thread. - **A list change answers before the snapshot is written.** Once the lists are parsed and merged, the new rules start answering ("filter active (new rules from the merge; writing the snapshot)" in the log), and the snapshot's files are written after that; the finished snapshot then takes over with the same lookup index. If writing it fails (a full disk), the previous rules answer again, so a node never serves rules it has no snapshot of. Cluster replicas get the finished snapshot, as before. - Thread count, `[filter] compile_threads`: the default, `0`, uses half the cores (between 1 and 4, so 2 on a Pi 4) when nothing is filtering yet, so blocking starts quickly on a first start. Once a filter is serving, a recompile (list refresh, reload) uses **one** thread: it takes longer, but nobody waits for it, and more threads compete with queries for cores and memory bandwidth even at the lowest priority. Set a number to use it for every compile. - **Regex rules are capped.** `[filter] max_regexes` (default 1,000) is the most regex rules a snapshot keeps; the ones past it, in list order, are left out, counted per list on the Lists page ("N regex rules left out"), and named in the log (the first 20) and by `telltale lists compile`. Every query that no exact or suffix rule settles is tried against all the regexes, about a microsecond per thousand (200 cost 0.4 µs at the median, 1,000 cost 1.1 µs). The matcher's automaton does not scale past a few thousand patterns that share a prefix: in a synthetic test, 1,500 cost 1.6 µs per query and 41 MiB, but 2,000 grew past 5 GiB of memory within two minutes. Raise the limit only after measuring your own lists (`matcher_bench`), and prefer plain domain rules where a list offers both. - **Power cuts.** A snapshot's files and its directory are synced to disk (`fsync`) before it counts as published, so a power cut can't leave a snapshot whose files are short; on an SD card that adds a few hundred milliseconds to a large compile, off the query path. `[filter] fsync = false` skips it. If the newest snapshot can't be opened at a start anyway (a snapshot made with `fsync = false` before a crash, say), it is renamed `.broken`, the next newest is used, and the `.broken` directory is deleted after the next successful compile. Without an older snapshot, the resolver starts unfiltered until lists compile. - Memory stays bounded. List entries are sorted within `[filter] compile_memory` (default `"128MiB"`), and anything beyond that spills to temporary files in the snapshot directory. Each list's text is read only while it's being parsed. - **Sizing memory limits.** Steady state, the filter costs about 20 bytes per blocked name on top of a ~17 MiB base: 30–35 MiB with HaGeZi Pro + OISD big + StevenBlack (430k names), about 70 MiB with 2.7M names. Compiling needs much more for a few seconds: about 110 MiB peak for those three lists, and about 300 MiB for 2.7M names. If you set a container memory limit, leave room for that peak (at least 256 MiB for list sets over ~1M names), or the process can be killed mid-compile. Limiting compile memory automatically is planned (ADR-025). - Size and speed: about 9 bytes per blocked name. On a Raspberry Pi 4, a recompile of 2 million names on one thread (the default) answers with the new rules after about 4.7 s and finishes writing the snapshot at about 12.8 s. A laptop-class machine makes 2.7M names answer in about 2 s on one thread. ```sh telltale lists compile -c telltale.toml # compile now and show per-list numbers # snapshot 1 in /var/lib/telltale/snapshots/1 # names 2597518 (subtree 2597518, exact 0, subdomains 0), regexes 0, modifier rules 0, list sets 7, $badfilter removed 0 # 8.98 bytes/name; 4.56s total (parse 2.03s, merge 2.49s, tables 9.10ms) on 1 thread(s); 0 sort runs spilled # hagezi-tif entries 2376001 unique 2213895 unsupported 0 invalid 0 # hagezi-pro entries 227420 unique 102069 unsupported 0 invalid 0 # oisd-big entries 244348 unique 55809 unsupported 0 invalid 0 ``` `unique` counts the names no other list has. A list with few unique names adds little beyond your other lists. The **Lists** page and `GET /api/v1/lists` show the same per list: `unique`, `hits` (queries it decided on this node since it started), and `overlap` (the lists it shares names with, and how many). A list that's 95% or more inside another is marked, and one with no hits is marked "none yet". ### Blocking A query is checked against the filter after local records and before the cache, so a blocked name never reaches an upstream: ``` $ dig @192.168.1.53 ads.example.com ;; ->>HEADER<<- opcode: QUERY, status: NOERROR ; EDE: 15 (Blocked): (blocked by list hagezi-pro) ads.example.com. 60 IN A 0.0.0.0 ``` How blocked names are answered is set per group (the client's highest-priority group decides): ```toml [[group]] name = "kids" block_mode = "null_ip" # default: A → 0.0.0.0, AAAA → ::, other types → no records # or "nxdomain", "nodata", "refused", "custom_ip" block_ips = ["192.168.1.2", "fd00::2"] # for "custom_ip", e.g. a "blocked" page block_ttl = 60 # how long clients may cache the block ede = "filtered" # "blocked" (EDE 15, default) or "filtered" (EDE 17: parental controls) ede_text = true # name the list in the EDE text (set false to hide list names) ``` `null_ip` is the default, as in Pi-hole and Technitium. Apps treat it as "unreachable" and give up quickly, while NXDOMAIN makes some apps retry or fall back to another resolver. **CNAME inspection:** if an answer leads through a CNAME to a blocked name (`www.microsoft.com` → `…edgekey.net` → `….akamaiedge.net`), the answer is replaced with the client's block answer, with EDE text `CNAME target blocked by list …`. This also applies to answers served from the cache, which is shared by every group, so one group's policy never leaks into another's. **Pause:** blocking can be paused for everyone or for one group, for a set time, and resumes on its own. Pause and resume through the API or `telltale ctl pause --minutes 10` (`ctl resume`); the metric `telltale_filter_paused_until_seconds{group}` shows active pauses. When rules disagree, the most important one wins: | Wins | Rule | Example | |---|---|---| | 1 | important allow | `@@\|\|cdn.example.com^$important` | | 2 | important block | `\|\|ads.example.com^$important` | | 3 | allow (allowlists, `@@` rules) | `@@\|\|cdn.example.com^`, or a `kind = "allow"` list | | 4 | block | everything else | Among rules of the same kind, the one reported is an exact-name rule first, then the longest matching domain, then a regex, then the list listed first in the config. Lookups take well under a microsecond: about 0.3 µs typically and under 1 µs at p99 on an x86 server with 2.6M blocked names. They use an in-memory index built from the snapshot (about 10 bytes per name, on top of the snapshot's 9). A new snapshot is served the moment it's loaded, and the index is swapped in about a second later, without pausing queries. Memory with HaGeZi Pro + TIF + OISD big + StevenBlack + AdGuard (2.7M names): about 85 MiB in total; HaGeZi Pro alone (227k names): about 10 MiB. Metrics: `telltale_queries_total{status="blocked"}` (including CNAME blocks), `telltale_filter_paused_until_seconds{group}`, `telltale_filter_lookup_index_bytes` (0 while the index is being built), `telltale_filter_snapshot_version`, `telltale_filter_rules`, `telltale_filter_compile_seconds`, `telltale_list_entries{list}`, `telltale_list_source_bytes`, `telltale_list_last_success_timestamp_seconds`, `telltale_list_last_change_timestamp_seconds`, and `telltale_list_fetch_consecutive_failures`, each labeled `{list}`. To alert on a list that has failed for two days: ``` time() - telltale_list_last_success_timestamp_seconds > 172800 ``` ### Lists that stopped changing A list whose maintainer gave up can keep answering HTTP 200 with the same content forever: it downloads fine, so nothing fails, but it stops keeping up. A URL list whose content hasn't changed in 30 days is called **stale**: - the **Lists** page says *unchanged for N days*, the health icon shows `list_stale` (degraded), and the alert rule `list_stale` can tell you; - `GET /api/v1/lists` has `staleDays` for it. Change the limit for every list, or for one that rarely changes on purpose (0 = never stale): ```toml [filter] stale_after_days = 30 [[list]] name = "my-small-list" url = "https://example.com/rarely-changes.txt" stale_after_days = 0 ``` Files and inline rules change when you change them, so they're never stale. The days count from the last time the content changed (identical downloads don't count), which is also `telltale_list_last_change_timestamp_seconds`. ### Trying a list first (shadow mode) Not sure a list is safe for your network? Add it in **shadow mode**: it's downloaded and checked like any other, but never blocks. ```toml [[list]] name = "hagezi-ultimate" url = "https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/wildcard/ultimate.txt" mode = "shadow" # enforce (the default) or shadow ``` - **What you see:** the **Lists** page marks it *shadow*, and **Would have blocked** shows, per shadow list, how many queries it would have blocked, from how many devices, the names most often, and the last one. When those are names you want gone, change `mode` to `enforce` (or edit the list on the Lists page). `GET /api/v1/analytics/shadow-lists`, the MCP tool `shadow_lists`, and `telltale_list_shadow_hits_total{list}` have the same. - **How:** shadow lists sit in the compiled snapshot but in no group's enforcing set. For each query answered normally, the telemetry thread asks the filter again with the device's group's lists plus its shadow lists; a block by a shadow list is counted. It respects groups (a shadow list only counts for groups that would use it), allow lists, and `$client`-style rules. It never runs on the query path; with no list in shadow mode it does nothing. A very busy server may skip some (dropped events are counted), never slow down. Counts start at each restart. In **explain**, a shadow list's rules say *shadow*. - **Likely over-blocking** (on the same page, `GET /api/v1/analytics/overblocking`, the MCP tool `overblocking_suspects`): blocked names that someone seems to want. *Allowed soon after* counts devices that got the name answered within 10 minutes of a block (you paused blocking, or allowed it); *retry bursts* count devices asking for it 10 or more times within a minute while it was blocked (apps that break retry; ad libraries do too, so it's the weaker sign). With the lists that blocked them: candidates for an allowlist, or a sign a list is too aggressive. - Names follow the query log's privacy level (hashed at 1 and above); devices aren't counted at 2 and above. - Like every list, a shadow list is checked once the snapshot is compiled (a few seconds after start or a change): queries answered before that aren't counted. `telltale explain` marks its rules with `~`. ## Groups and devices Decide which lists apply to which devices: ```toml [[group]] name = "kids" lists = ["hagezi-pro", "family-extra"] # omit `lists` to use every list priority = 10 # highest-priority group's settings win [[group]] name = "default" # optional: what unknown devices get lists = ["hagezi-pro"] # (without it, unknown devices get every list) [[client]] name = "Kids tablet" match = ["aa:bb:cc:dd:ee:01", "192.168.1.50", "id:kids-tablet"] groups = ["kids"] [[client]] name = "Office" match = ["10.0.5.0/24"] # no groups: its network's group, else "default" ``` - A device in several groups gets every list of all of them. Other settings, such as block mode, come from its highest-priority group. - **How a query's device is recognized,** first match wins: client ID (`id:…`, from the DoH URL path `/dns-query/` or the DoT/DoH server name `.dns.example.com`) → MAC address, from a trusted router's EDNS option or from the kernel neighbor table → exact IP → the most specific CIDR → `default`. - **MAC addresses** survive DHCP changes and IPv6 privacy addresses, which makes them the most dependable key for a home network. While any `[[client]]` matches by MAC, TelltaleDNS reads the kernel's neighbor table (ARP and IPv6 NDP) every 60 s (`[clients] neighbor_refresh_secs`); without one it isn't read, and `telltale_neighbors` stays 0. The table only holds devices on the node's own subnets, and only with host networking (as on a Pi): in a container's bridge network or a Kubernetes pod, every query appears to come from the gateway. Device recognition also gets MAC addresses from router integrations and mDNS, with or without the table. - If your router forwards queries with dnsmasq's `add-mac`, trust it explicitly. Any device could send that option and impersonate another, so it's ignored by default: ```toml [clients] trust_edns_mac_from = ["192.168.1.1/32"] ``` - `$client=` rules match the device's name (`$client='Kids tablet'`), its client ID, or its IP or CIDR. - A group can also send its questions to its own upstream servers (for example, a family-filtering resolver): see [Upstream servers per group](#upstream-servers-per-group). - Changes apply on reload (`SIGHUP`). Metric: `telltale_neighbors` (entries in the neighbor table). ### Groups for your networks (VLANs) A group can own networks: every device on them belongs to it, with no per-device setup. That makes groups a way to see *which kinds of devices* make which traffic: ```toml [[group]] name = "IOT" networks = ["192.168.2.0/24"] color = "#22c55e" # optional: its color in charts and chips [[group]] name = "Surveillance" networks = ["192.168.7.0/24"] [[group]] name = "Trust" networks = ["192.168.10.0/24"] ``` - **Which group a device gets:** the group of the most specific matching network. A `[[client]]` entry with its own `groups` still wins. Naming a device (in the UI or with a `[[client]]` entry without `groups`) keeps it in its network's group. Devices outside every network are in `default`. - **Filtering doesn't change** unless you give the group `lists`: a group without `lists` uses every list, like `default`. - **A network belongs to one group:** the same CIDR in two groups is a configuration error. Nested networks are fine; the most specific wins. - **Where it shows:** - the dashboard's **Traffic by group** chart (in the Advanced view), and a group selector for its top lists; - each group's card on the **Groups** page: networks, queries and block rate over 24 hours, devices this hour, top names and top blocked; - a group chip on every query-log row, and a **Group** filter; - `group=` on `/api/v1/stats/top` and `/api/v1/queries`, and `byGroup` / `blockedByGroup` in timeseries buckets; - the metric `telltale_group_queries_total{group,status}`. - **In a cluster**, groups and networks are shared configuration, so every node attributes devices the same way. ### Upstream servers per group Each group can send its questions to its own `[[upstream_group]]` instead of `default`: a family-filtering resolver for the kids, a privacy resolver for the trusted network, or the ISP's resolver for streaming boxes that care where they're answered from. Make the upstream groups first, from [presets](#upstream-presets) or by hand: ```sh telltale presets show cloudflare-family --proto tls --group family >> telltale.toml telltale presets show quad9 --proto tls,https --group privacy >> telltale.toml ``` Then point groups at them: ```toml [[group]] name = "Kids" networks = ["192.168.20.0/24"] upstreams = "family" priority = 10 [[group]] name = "Trust" networks = ["192.168.10.0/24"] upstreams = "privacy" ``` - **In the UI:** Groups → Edit → **Upstream servers**. *(default)* means the `default` upstream group. The group's card shows the choice. - **What wins,** most specific first: 1. a `[[route]]` for a domain (`match_suffix`), such as `home.arpa` to your router, applies to every group, so local names keep working; 2. a `[[route]]` with `match_group` naming the group (for example, only its `TXT` questions); 3. the group's `upstreams`; 4. `default`. - **A device in several groups** (a network group plus a `[[client]]` group) goes where its highest-priority group with `upstreams` says. - **Caching is per upstream group.** An answer one group got from its resolver is never served to a group that uses another, so a family resolver's answers stay with the family group. - **Removing an upstream group** that a group (or a route) still uses is refused; pick other upstream servers for that group first. - No cost on cached answers: the choice is made only when a question goes upstream. ### Example: a home with VLANs Networks, lists, schedules, and upstream servers combine. Each VLAN gets its own rules, and every chart and log line shows which one a question came from: ```toml [[group]] name = "Trust" # laptops and phones networks = ["192.168.10.0/24"] lists = ["hagezi-pro"] upstreams = "privacy" [[group]] name = "Kids" networks = ["192.168.20.0/24"] lists = ["hagezi-pro", "family-extra"] blocked_services = ["tiktok", "fortnite"] safe_search = true schedules = ["bedtime"] upstreams = "family" priority = 10 [[group]] name = "IoT" # TVs, plugs, cameras networks = ["192.168.30.0/24"] lists = ["hagezi-pro", "smart-tv"] rebinding_protection = true [[group]] name = "Guest" networks = ["192.168.40.0/24"] lists = ["hagezi-light"] block_mode = "nxdomain" ``` - IoT has no `upstreams`, so it uses `default`. - To see the effect, filter the query log by group, or ask `from -24h | where group = Kids | top 10 upstream` (see [Analytics in one query](#analytics-in-one-query-vqlog)). - Devices must reach TelltaleDNS **with their own addresses** for their network to count: route the VLANs to it rather than through a forwarder that rewrites them (see [Seeing real client IPs](#seeing-real-client-ips)). ### Blocked services Block a whole service for a group with one setting, without finding its domains: ```toml [[group]] name = "kids" networks = ["192.168.20.0/24"] blocked_services = ["tiktok", "roblox", "fortnite"] ``` - `telltale services list` (or `GET /api/v1/services`, or the Groups page) shows every service with the domains it blocks: social networks (TikTok, Instagram, Snapchat, ...), video and music (YouTube, Netflix, Twitch, Spotify, ...), games (Roblox, Fortnite, Minecraft, Steam, Xbox, PlayStation, Discord), messaging and AI chat. - A service blocks its domains and everything under them, for that group only. A group that uses every list still doesn't get other groups' services. - Blocks say which service did it: the query log shows the service, and the EDE text reads `blocked service TikTok`. - In the UI: **Groups → Manage groups → Edit → Blocked services**. - The catalog is part of the binary (`presets/services/*.toml`); services change with updates. To block one service's extra domain, add a quick rule. ### Rewrites Answer a name differently for a group, for example a split-horizon name or a test server: ```toml [[group]] name = "default" [[group.rewrite]] domain = "nas.example.com" # this name... answer = "192.168.1.10" # ...answered with this address (A, or AAAA for IPv6) [[group.rewrite]] domain = "*.dev.example.com" # every name under dev.example.com answer = "10.1.1.1" [[group.rewrite]] domain = "tv.example" answer = "cdn.example.net" # answered as a CNAME to that name, with its addresses ``` - An exact name wins over a wildcard; the most specific wildcard wins over others. A wildcard covers names under the domain, not the domain itself. - An address rewrite answers the other address family and other record types with no data. - Blocks still win: a blocked name isn't rewritten. For a name everyone should get, use [local records](#local-records) instead. ### DNS64 For IPv6-only devices behind a NAT64 gateway: names without an IPv6 address get one made from their IPv4 address (RFC 6147), so the gateway can reach them. ```toml [[group]] name = "ipv6-only" networks = ["2001:db8:64::/64"] dns64 = true # dns64_prefix = "64:ff9b::/96" # the default (well-known prefix); your NAT64's /96 otherwise # dns64_exclude = ["2001:db8:bad::/48", "198.51.100.0/24"] ``` - An AAAA question with no AAAA answer is asked again as A, and each A record becomes an AAAA in the prefix (`192.0.2.33` → `64:ff9b::c000:221`); CNAMEs come along. Names that have real AAAA records are left alone. - Only for the group's devices, and never for clients that set CD (they validate DNSSEC themselves, and made-up records don't validate). - Check it with `dig AAAA ipv4only.arpa`: through DNS64 it answers `64:ff9b::c000:aa` and `64:ff9b::c000:ab`. - Addresses that can't work through NAT64 (`0.0.0.0/8`, `127.0.0.0/8`, `169.254.0.0/16`) aren't turned into AAAA. - **Exclusions** (`dns64_exclude`, RFC 6147 §5.1.4): IPv6 addresses in these networks count as missing, for names whose IPv6 addresses your devices can't reach (a broken tunnel, a provider's unreachable range); the name gets made-up AAAA records instead, or keeps its other real ones. IPv4-mapped addresses (`::ffff:0:0/96`) are always excluded. IPv4 networks listed here are never turned into AAAA, so those destinations stay unreachable from IPv6-only devices. - **Reverse lookups** for addresses in the prefix follow the IPv4 address: `dig -x 64:ff9b::c000:221` answers with a CNAME to `33.2.0.192.in-addr.arpa` and that name's PTR (RFC 6147 §5.3.1). Private IPv4 addresses inside the prefix get NXDOMAIN locally, as their in-addr.arpa names would. ### No IPv6 addresses for a group The opposite of DNS64: a network whose IPv6 is broken or half set up (pages hang, then fall back to IPv4). Answer the group's IPv6 (AAAA) questions with nothing, so its devices use IPv4 at once: ```toml [[group]] name = "guest" networks = ["192.168.50.0/24"] filter_aaaa = true ``` - AAAA questions get NOERROR with no records, from TelltaleDNS itself (no upstream asked); A and every other type are unchanged. The query log shows them as `special`. - Your [local records](#local-records) and zones still answer AAAA (you configured those addresses), and blocked names stay blocked. - HTTPS records (type 65) can still carry IPv6 hints, which some browsers use; this setting leaves them alone. - Not with `dns64` (which makes AAAA answers on purpose). In the UI: **Groups**, edit the group, *No IPv6 addresses* (Advanced view). ### Rebinding protection A website can make your browser attack devices on your network by pointing its own name at a private address (DNS rebinding). Turn on the protection per group: ```toml [[group]] name = "default" rebinding_protection = true # refuse private answers for public names rebinding_allow = ["plex.direct", "home.example.com"] # names allowed to answer private addresses block_answer_ips = ["203.0.113.0/24"] # answers in these networks are refused for any name ``` - With `rebinding_protection`, answers in private (10/8, 172.16/12, 192.168/16, fc00::/7), loopback, link-local, CGNAT (100.64/10), and unspecified ranges are blocked, with the block answer and the reason in the query log. Your [local records](#local-records) and domains with a `[[route]]` (conditional forwarding to your router or AD server) are never affected. - `block_answer_ips` refuses answers in any network you list, whatever the name (a known-bad range, or your own WAN address). - Cached answers are checked too, and a paused group isn't filtered. ### Safe search Make search engines show only safe results for a group, whatever the browser's settings: ```toml [[group]] name = "kids" networks = ["192.168.20.0/24"] safe_search = true youtube_restrict = "strict" # or "moderate", or "off" to leave YouTube alone ``` - Google (every country domain), Bing, DuckDuckGo, Yandex, and Pixabay answer as a CNAME to their own safe-search service (`forcesafesearch.google.com`, `strict.bing.com`, ...), with that service's addresses, so the engine enforces it. YouTube gets Restricted Mode (`restrict.youtube.com`, or `restrictmoderate.youtube.com`). - HTTPS records for those names get no data, so browsers use the safe-search addresses rather than hints for the normal ones. - It applies to the group's devices only; the table is part of the binary (`presets/safesearch.toml`). - Browsers using their own encrypted DNS bypass it (as they bypass all filtering): TelltaleDNS answers `use-application-dns.net` with NXDOMAIN so Firefox turns its own DoH off. - In the UI: **Groups → Manage groups → Edit → Safe search**. ### Schedules Change a group's filtering by the time of day and the day of the week: block everything at bedtime, or add a list or a service only at certain times. ```toml [[group]] name = "kids" networks = ["192.168.20.0/24"] schedules = ["bedtime", "school-hours"] [[schedule]] name = "bedtime" action = "block_all" # everything is blocked during the windows tz = "Europe/Berlin" # default: the system's time zone window = [ { days = ["sun", "mon", "tue", "wed", "thu"], start = "21:00", end = "07:00" }, { days = ["fri", "sat"], start = "22:30", end = "08:00" }, ] [[schedule]] name = "school-hours" action = "block_services" # or "enable_lists" with `lists = [...]` services = ["tiktok", "youtube", "roblox"] window = [{ days = ["weekdays"], start = "08:00", end = "15:00" }] ``` - **Windows** are weekly: the days they start on (`mon` … `sun`, `weekdays`, `weekends`, `daily`) and local `HH:MM` times. An end at or before the start runs past midnight, so `21:00`–`07:00` on Sunday covers Sunday night into Monday morning. - **`block_all`** blocks every name for the group (the answer says `blocked by schedule bedtime`). Quick allow rules still win, so a homework site can stay reachable, and local names (`nas.home.arpa`) still answer. Pausing a group's blocking pauses its schedules too. - **`enable_lists`** and **`block_services`** add lists or services during the windows. A list a schedule turns on is off outside its windows for every group, even one that uses every list, unless a group names it in its own `lists`. - Schedules are checked every 15 seconds, so a window starts or ends within 15 s of its time. No query does any time maths, and nothing is recompiled. - The Groups page shows each group's schedules and which are **on now**; the query log shows blocks as **schedule**. - **In the UI:** **Groups → Schedules** adds, changes, or removes schedules (windows one per line, like `weekdays 21:00-07:00`), and a group's editor picks the schedules it follows. Through the API: `PUT`/`DELETE /api/v1/schedules/{name}` with the same fields as `[[schedule]]` (agents need `config:write:groups`, and AI agents can plan one with the MCP tool `plan_set_schedule`). ### Names from your router (UniFi, OPNsense) TelltaleDNS doesn't hand out addresses: your router's DHCP does that, and TelltaleDNS can read its client list, so devices show up by name: ```toml [[router]] name = "udm" type = "unifi" # UniFi OS consoles (UDM, UCG, Cloud Key) and the Network application url = "https://192.168.1.1" # the classic Network application: https://controller:8443 api_key_file = "/etc/telltale/unifi-key" # UniFi Network 9+ (Settings → Control Plane → Integrations) # username = "telltale" # or a local read-only user... # password_file = "/etc/telltale/unifi-pass" # site = "default" tls_ca = "/etc/telltale/udm.pem" # the console's own certificate (self-signed) tls_server_name = "unifi.local" # the name in it (UniFi consoles name themselves unifi.local) [[router]] name = "firewall" type = "opnsense" url = "https://opnsense.lan" api_key_file = "/etc/telltale/opn-key" # System → Access → Users → API keys api_secret_file = "/etc/telltale/opn-secret" ``` - UniFi: the connected clients, named by the alias you gave them in UniFi, else their host name. OPNsense: the DHCP leases (Dnsmasq, Kea, or ISC, whichever is running). - A self-signed console: save its certificate (`openssl s_client -connect 192.168.1.1:443 udm.pem`) as `tls_ca`. Its certificate names `unifi.local`, not the address you reach it at, so set `tls_server_name` to the name in it (`openssl x509 -in udm.pem -noout -ext subjectAltName` shows it): the console is then verified, and only that certificate is trusted. `tls_insecure_skip_verify = true` also works, but then anything on your network could pose as the console and receive the API key. - Read every `interval_secs` (300); a router that doesn't answer is asked again after 15 seconds and keeps its last list meanwhile. - Names you give devices in TelltaleDNS win; then the routers'. `GET /api/v1/dhcp/leases` lists them (`source = router`). - Credentials stay in files, out of the configuration. Changes to `[[router]]` need a restart. - In a cluster, only one node needs to read the router (or hear mDNS). Every node asks the others for the device names they know, once a minute, so a resolver pod without the router's key names devices the same way the primary does. ### Names devices announce (mDNS) Phones, speakers, printers, and NAS boxes announce themselves on the local network (`Kitchen-HomePod.local`). TelltaleDNS can listen and use those names for devices you haven't named: ```toml [clients] mdns = true ``` - Listen only: TelltaleDNS never sends mDNS. It shares port 5353 with avahi or the system's responder. - It has to hear the LAN: a native install, the Pi bundle, or a pod with host networking. In a bridge or pod network nothing arrives. - Names you give devices win, then the routers' DHCP leases, then mDNS. Names are lowercase, as mDNS compares them. At most 4096 devices are remembered (until a restart). - `GET /api/v1/dhcp/leases` lists them with `source = mdns` (no MAC), and **Clients → Discovered** shows every device found by a router or mDNS. Naming one from there (or from any address in the UI) suggests the name it gave. - Changing it needs a restart. ### Naming devices in the UI Click a device's address anywhere (the dashboard's top clients, the query log, the live view, Clients) and choose **Name this device…** or **Add to group…**. The name shows everywhere at once, including on past queries: names are looked up when data is read, never written into the query log, so renaming relabels history and forgetting a device brings the address back. Devices named this way are stored in `state.db` (next to users and the audit log) and merged with `[[client]]` entries from the config files. The files win: a device defined in a file is read-only in the UI, and the UI can't reuse its name. If a stored device stops being valid (say its group was removed from the file), TelltaleDNS logs why and runs with the file config alone until it's fixed. Operators and admins can name devices; viewers see the names. The same through the API, for scripts and agents: ```sh curl -X PUT 'https://dns.example.com/api/v1/clients/Living%20room%20TV?dryRun=true' \ -H "authorization: Bearer $TOKEN" -H "content-type: application/json" \ -d '{"match": ["192.168.1.42"], "groups": ["default"]}' # what would change; nothing is saved ``` Without `dryRun` the change is saved, applied, and audited. `GET /api/v1/clients` returns the config version as `ETag`; send it back as `If-Match` to refuse the write if someone changed the config meanwhile (412). An `Idempotency-Key` header makes retries safe: the same key replays the first answer for 24 hours. To rename, send the new `name` in the body; `DELETE /api/v1/clients/{name}` forgets a device. ### What is this device? TelltaleDNS guesses what each address is, so a new one isn't just a number. It uses what it already sees: - the maker of the device's network card: its MAC address's vendor in the IEEE registry (from the neighbor table or your router); - the name the device announces (to your router's DHCP, or over mDNS); - the domains it talks to (a Roku asks `roku.com`, a Sonos speaker `sonos.com`). These are scored against a catalog of common home devices that ships with TelltaleDNS (phones, TVs and streamers, speakers, cameras, plugs, consoles, printers, NAS boxes, network gear, and more). The Clients page then says, under the device: "Looks like a **Roku player** (likely) · Roku". **Likely** and **possibly** say how sure it is; when nothing fits, only the maker is shown. **Why?** lists the evidence: the domains that matched, the vendor and MAC prefix, the name, and a close second guess if there is one. - **Naming:** "Name this device…" fills in the guess ("Roku player") when the device hasn't told your router a name, so you only add "living room". A guess never becomes a name by itself. - **Groups:** give a group the kinds of device that belong in it, and devices of that kind that aren't in it yet get it suggested ("iot suggested"); "Add to group…" pre-selects it. Nothing moves on its own. ```toml [[group]] name = "iot" device_classes = ["camera", "doorbell", "plug", "bulb", "hub", "thermostat", "vacuum"] ``` A kind belongs to one group at most. - **Wrong guess?** Name the device and set **What it is** (`[[client]] kind = "camera"`, or `"unknown"` to stop guessing). That always wins and shows as set rather than guessed. - **More devices:** `[identify] signatures_file = "/etc/telltale/devices.toml"` adds signatures in the catalog's own shape (`presets/devices.toml` in the source): an `id` the catalog uses replaces that signature, and `enabled = false` on it removes it. ```toml [[device]] id = "our-bike" name = "Exercise bike" class = "appliance" vendors = ["Acme Fitness"] # words in the MAC vendor hostname_patterns = ["BIKE-*"] # globs on the names it announces domains = [{ name = "acmefit.example", weight = 1.0 }] ``` How it works: every 10 minutes, on a thread of its own at background priority, each node scores the devices it saw in the last day (`[identify] max_clients`, 1,024 by default), from what it already keeps in memory; for a device it doesn't hold, it reads the query log (a few devices per pass). The score is 60% domains, 25% maker, 15% name: **likely** from 0.6, **possibly** from 0.35. Results are kept in `/devices.json` across restarts. In a cluster each node identifies the devices it sees, and the most confident guess per device is shown. It never changes how a query is answered. Limits: phones use private Wi-Fi addresses, so their maker is unknown and the domains and name decide. Behind a router or proxy that hides client addresses, many devices look like one. It's off at query-log privacy level 1 and above, where names are hashed, and with `[identify] enabled = false`. Through the API: `identity` on each device of `GET /api/v1/clients`, `GET /api/v1/clients/identities` for every device seen, and `GET /api/v1/clients/{address or name}/identity` with the evidence. Agents: `identify_device`, and `identity` in `get_client_profile`. New-device alerts and anomaly findings say what the device looks like. Metrics: `telltale_devices_by_class{class}`. The MAC vendor table is built from the IEEE Registration Authority's public registry by `presets/build-oui.py`. ## Quick rules A quick rule allows or blocks a site, and everything under it, for one device, a group, or everyone. It can last for a while or until you remove it. Some uses: - unblock a game's server on one phone for an evening; - block a video site on the kids' tablets until tomorrow; - allow something a list blocks by mistake while you report it. **Making one:** - In the **query log**, press **Why?** on a query and use **Make a quick rule**. The site, the device, and its group are filled in. - On the **Quick rules** page (Filtering & DNS), which also lists what's in effect with a countdown, who made each rule, and a Remove button. **How they decide:** - Quick rules decide **before any list**, on the next query. The lists aren't recompiled. - When several match, a rule for the **device** beats a rule for its **group**, which beats a rule for **everyone**. - Within one of those, the more specific site wins (`docs.example.com` over `example.com`), then allow over block. - They also decide CNAME targets, so blocking `videos.example` catches a name that's a CNAME to it. - A group whose blocking is paused gets no blocks from quick rules either. **Expiry and history:** - A rule with an end time stops applying at that moment on every node, by each node's clock (the Cluster page flags clock drift). - Within seconds it's removed, and the audit log records `rule.expire`. - The query log shows decisions as **quick allow** or **quick block**, with the rule's note. **Limits** (DNS can only do so much): - DNS sees **sites, not pages**: blocking a page means blocking its whole site. - An app often uses many sites, some shared with other apps. Unblocking one game server is precise; blocking a whole app is best done with a list. - A rule follows a device only as well as TelltaleDNS **recognizes** it. Give the device a fixed address (a DHCP reservation), or turn off its private Wi-Fi address for your network. - Devices keep answers they already have for a few minutes, so a change can take that long to show on the device. Unblocking needs no cache flush, because blocked answers are never cached here. - **Mobile data, a VPN, or a hard-coded DNS server** go around TelltaleDNS. To close that, have your router block outgoing DNS (ports 53 and 853) to anything but TelltaleDNS. Quick rules aren't parental controls: there's no screen time and no content categories. In a config file (useful with GitOps), `[[rule]]` takes the same fields. The API can't remove these rules. ```toml [[rule]] id = "kids-videos" action = "block" # or "allow" domain = "videos.example.com" # and its subdomains groups = ["kids"] # or devices = ["Mom phone", "192.168.1.40"]; neither = everyone expires = "2026-10-06T07:00:00Z" # optional (RFC 3339) note = "school night" ``` Through the API (operators, or agent tokens with `config:write:rules`): ```sh curl -X PUT https://dns.example.com/api/v1/rules/mom-game \ -H "authorization: Bearer $TOKEN" -H "content-type: application/json" \ -d '{"action": "allow", "domain": "game.example.com", "devices": ["Mom phone"], "forMinutes": 120, "note": "Mom'"'"'s game"}' curl https://dns.example.com/api/v1/rules -H "authorization: Bearer $TOKEN" # what's in effect ``` `?dryRun=true` shows what would change and how many recent queries it affects. `DELETE /api/v1/rules/{id}` removes a rule. ## Why was it blocked? (explain) `telltale explain` shows what happens to a name for a given device, and why. It lists: - who the device is recognized as, and its groups; - every rule in every list that matches, with its file line, in precedence order; - which rule decides; - where the query would be forwarded. It follows the same steps as the server, in the same order: access checks, local records and authoritative zones, quick rules, a schedule that blocks everything for the group, list `$dnsrewrite` rules (which win over blocking), the lists, then the group's `[[group.rewrite]]` entries and safe search. When one of the later steps answers the query instead of a list, the outcome and summary say which. ```sh telltale explain ad.doubleclick.net --client 192.168.1.50 -c telltale.toml # ad.doubleclick.net A from 192.168.1.50 # client tablet (identified by ip); groups: kids # outcome ALLOWED: allowed by list family-allow; resolved normally # rules snapshot 1, in precedence order (* decides, - list not used by this client, ~ shadow list: counted, never blocks) # * allow family-allow doubleclick.net (and subdomains) # family-allow:1 @@||doubleclick.net^ # - block stevenblack ad.doubleclick.net (and subdomains) # stevenblack:7102 0.0.0.0 ad.doubleclick.net # route upstream group default (default) ``` - Options: `-t AAAA` for another query type, `--mac aa:bb:cc:dd:ee:ff` or `--client-id` to explain for a device recognized that way, and `--json` for the same data as JSON (the format the API's `GET /api/v1/explain` returns). - `-` marks rules from lists the device's groups don't use, so you can see what another group would get. `!` after `allow`/`block` marks `$important` rules. - It reads the config and the data directory (the newest compiled snapshot and the stored lists), so it works whether or not the server is running. A server that hasn't loaded the newest snapshot yet may still be using the previous one. - Line numbers come from the stored copy of each list. If a list was downloaded again after the snapshot was compiled, the output says so. - Explain covers the query name only. CNAME targets in an upstream answer are checked too when the server answers (see [Blocking](#blocking)); explain a target name to see its rules. ## Trying a change against yesterday's traffic Explain answers for one name. A simulation answers for a whole change: before you save it, the queries already in the query log are decided again with the change and without it, and only the differences are reported. - **In the UI:** the quick-rule form, the device drawer ("Add to group…"), and the checked preview of a list, group, upstream, or schedule have a **What would this have done?** button. It shows four counts (newly blocked, newly allowed, a changed route, a changed answer), the names behind each with the list or rule that decided, and the devices and groups. A name links to the query log. Nothing is saved until you save the change. - **Any dry run:** add `simulate` to a change's `?dryRun=true`. The value is a window, or `true` for the default one. The response gains `simulation`: ```sh curl -X PUT 'https://dns.example.com/api/v1/rules/kids-tiktok?dryRun=true&simulate=24h' \ -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \ -d '{"action":"block","domain":"tiktok.com","groups":["kids"]}' # "simulation": {"rows": 183204, "newlyBlocked": {"queries": 412, "devices": 2, # "topNames": [{"name": "www.tiktok.com", "queries": 300, "devices": 2, "list": "quick rule: tiktok.com"}]}, ...} ``` `simulateUntil` moves the end of the window back (`-24h`, or an RFC 3339 time). A change that can't affect an answer, such as an alert rule or the exclusions, answers `"applicable": false` instead of replaying anything. - **A whole configuration:** `POST /api/v1/simulate` takes the shared sections as JSON, the way a Git commit's TOML becomes. `telltale config simulate` reads the TOML for you, so you can test a commit before pushing it: ```sh TELLTALE_TOKEN=... telltale config simulate shared.toml --window 24h --url https://dns.example.com:8053 # 183204 logged queries from 2026-10-08T12:00:00.000Z to 2026-10-09T12:00:00.000Z # QUERIES DEVICES # newly blocked 412 2 # ... ``` Node-local sections in the file are ignored. How it works, and its limits: - **Only the change is compiled.** The running filter is reused. Lists the change adds or edits are compiled on the side, in a temporary snapshot that is removed afterwards, and their matches are merged with the running ones by the usual precedence. A list being removed simply stops taking part. - **What is decided again:** who may query, local records and zones, quick rules (at each query's own time), schedules (likewise), list rewrites, the lists, the AAAA filter, group rewrites and safe search, and the upstream group a query would go to. Rate-limited, malformed, and dropped queries never reached those steps and count as unchanged. A pause of blocking isn't replayed: both sides are decided as if nothing were paused. - **Devices** are recognized again by address. A device that was recognized by its MAC address or client ID when it asked keeps the group its queries were logged with, and the result notes it. - **Bounds:** one simulation runs at a time per node (a second one gets `409 simulation_busy`, retry in a few seconds). Each node reads at most `[simulate] max_rows` queries, newest first, for at most `max_secs` seconds, at background priority; past either bound the answer says `partial`. The window is at most 7 days. - **Clusters:** every node replays its own log, and a controller also replays the logs its resolver pods ship to it. The counts are added up, and a node that didn't answer is listed in `missingNodes`. - **Privacy:** at query-log privacy level 1 names are stored hashed, so a change that depends on names can't be replayed (`"available": false, "reason": "privacy_level"`). Moving a device between groups still can: its queries count as unchanged when the new groups decide alike, else as undetermined. At level 2 and above nothing can be replayed. - DNS answering never waits for a simulation, and a simulation never asks an upstream or touches the cache. It's about decisions: it can't count names nobody asked, and it doesn't predict latency or what an upstream would answer. Settings: `[simulate]` (Settings → Query handling → Change simulation in the UI): `enabled`, `plans_by_default` (agents' plans simulate even when they don't ask, see [Plans and approval](#plans-and-approval)), `max_secs` (1 to 300, default 20), `max_rows` (at least 1,000, default 2,000,000), and `default_window` (default `24h`, at most `7d`). Metrics: `telltale_simulations_total{outcome}`, `telltale_simulation_duration_seconds`, `telltale_simulation_rows_total`. ## Who can query, and how often ```toml [access] # Default: private ranges, CGNAT/Tailscale (100.64/10), link-local, loopback. Everyone else: REFUSED. allowed_networks = ["192.168.0.0/16", "fd00::/8", "127.0.0.0/8"] [ratelimit] queries = 1000 # per client per window (bursts allowed up to this) window_secs = 60 action = "refused" # or "drop" exempt = ["127.0.0.0/8", "::1/128"] ipv6_prefix = 64 # a device's rotating IPv6 privacy addresses share one budget ``` TelltaleDNS is never an open resolver by default. If you widen `allowed_networks` to everything, a warning is logged at startup. Zone transfer requests (AXFR, IXFR) are answered `REFUSED` (with the extended error "Not Supported") and never forwarded: TelltaleDNS serves no zones to transfer. An old DNS server still trying to sync a zone through TelltaleDNS shows in the query log as `TYPE252` or `TYPE251`, status special. `[ratelimit]` applies on reload, with no restart. The default of 1,000 queries a minute per address suits homes where each device has its own address. A router or proxy that forwards for a whole network shows up as one address, and would be refused above about 17 queries a second: add it to `exempt`, or raise `queries`. - **Adjust it in the UI:** **Settings → Query handling → Rate limit** shows the limit in effect (from the config file, or changed here) and edits it: queries per window, the window, what a limited client gets (`refused` or `drop`), the addresses never limited, and in the Advanced view how clients are grouped (`ipv4_prefix`, `ipv6_prefix`). **Check** shows what the change does first. **Revert to the file** goes back to `[ratelimit]` in the configuration. A change replaces the whole section, so fields left out take the defaults, not the file's values, and the list of exempt addresses is replaced too (keep `127.0.0.0/8` and `::1/128`). - **Through the API:** `PUT /api/v1/ratelimit/default` with the same fields as `[ratelimit]` (add `?dryRun=true` to preview), and `DELETE` to revert; `GET /api/v1/config/entries?kind=ratelimit` shows the limit in effect and its source. Agents need `config:write:ratelimit`, and AI agents can plan a change with the MCP tool `plan_set_ratelimit`. On a node whose configuration comes from Git, the response says what to add to the repository. ### The advertised UDP size ```toml [dns] edns_payload = 1232 # 512 to 4096; needs a restart ``` This is the UDP payload size announced in EDNS(0) answers (RFC 6891), and the largest UDP answer sent to a client that announces at least as much; bigger answers are cut and flagged truncated, and the client retries over TCP. 1232 is the DNS Flag Day 2020 value and avoids IP fragmentation on almost every path. Raise it only on a network known to carry larger datagrams (jumbo frames, an IPv6-only network that doesn't fragment). The ceiling is 4096, the size of the UDP workers' send buffer. ### Which node answered (NSID) ```toml [dns] nsid = true ``` Answers carry this node's name in the NSID option (RFC 5001) when a query asks for it, so you can tell which node or pod answered when several share one address (a load balancer, resolver pods): ``` $ dig +nsid example.com @192.168.5.112 ; NSID: 74 65 6c 6c 74 61 6c 65 64 6e 73 2d 72 65 73 6f 6c 76 65 72 ... ("telltaledns-resolver-786d9bf5df-c6pr8") ``` - The name is `[node] name`, or the host name (a pod's name in Kubernetes). It's added only to answers whose query asked for NSID; others are unchanged. - Off by default: any client allowed to query sees the name. In a cluster, `[dns]` is shared, so one setting turns it on everywhere, and each node answers with its own name. ## Special names Handled before anything else (each can be turned off under `[special]`): | Name | Answer | Why | |---|---|---| | `localhost`, `*.localhost` | 127.0.0.1 / ::1 | RFC 6761 | | `*.invalid` | NXDOMAIN | RFC 6761 | | `use-application-dns.net` | NXDOMAIN | stops Firefox from silently switching to its own DoH (`block_firefox_canary`) | | CHAOS class (`version.bind`, …) | REFUSED | no fingerprinting (`refuse_chaos`) | | reverse lookups for private IPs | NXDOMAIN | never leaks your LAN layout to public resolvers, unless a local record or a `[[route]]` covers it (`private_ptr_nxdomain`) | `.local` names are passed through unchanged (many Active Directory domains use them; route them with `[[route]]`). ## Routing (conditional forwarding) ```toml [[upstream]] name = "home-router" url = "udp://192.168.1.1" [[upstream_group]] name = "lan" members = ["home-router"] [[route]] match_suffix = ["home.arpa", "168.192.in-addr.arpa"] # local names and reverse lookups upstream_group = "lan" ``` The longest matching suffix wins; routes can also match `match_qtype = ["PTR"]`. ### Stub zones (the zone's own servers) Forwarding asks a server to look the name up for you (recursion desired). A zone's own **authoritative** servers won't: they answer only for their zones, refuse everything else, and hand delegated sub-domains to other servers with a referral that a forwarder can't follow. For those, make the domain a **stub zone** (BIND's `static-stub`, Unbound's `stub-zone`): TelltaleDNS resolves names under it itself, starting at the zone's servers instead of the root, and follows their delegations. ```toml [[upstream]] name = "corp-stub" url = "recursive://" [upstream.recursive] stub_zone = "corp.example" stub_servers = ["10.0.0.53", "10.0.0.54"] # IP addresses, port 53 [[upstream_group]] name = "corp" members = ["corp-stub"] [[route]] match_suffix = ["corp.example"] upstream_group = "corp" dnssec_nta = true # internal zones are usually unsigned ``` - **The configured servers win**, whatever the public delegation for the zone says, so an internal view (split horizon) is what clients get. They're asked without recursion, the fastest first, and a slow one is hedged. - **Delegations are followed:** a sub-domain the servers delegate (`branch.corp.example` to `10.1.0.53`) is asked of its own servers, with the same bailiwick and glue checks as `recursive://`. - **Names outside the zone** that an answer leads to (a CNAME to a cloud service, a server name in another domain) are resolved from the root servers, like `recursive://`. The root never sees the stub zone's own names. - **Health checks** ask the stub's servers for the zone's SOA rather than the root's NS (which authoritative servers refuse, so a plain forward to them gets benched), and any answer except SERVFAIL counts as up, even REFUSED from servers that host only zones below it (a server that says it is failing for its own zone isn't up). A name under a delegated server that fails is a failed lookup, like with `recursive://`. - From the UI: **Names on my network → Send a domain to another server**, and tick **These servers only know this domain**. Through the API: `PUT /api/v1/forwards/corp.example` with `{"servers": ["10.0.0.53"], "stub": true}` (agents: `plan_forward_domain` with `stub: true`). - Unlike BIND's `type stub`, the zone's NS set isn't fetched from the servers: they're used as listed (`static-stub`). List every server you want asked. ## A warm cache after restarts With `[cache] persist = true`, TelltaleDNS writes its cache to `/cache.bin` when it stops, and reloads it when it starts. Upgrades and restarts then start with a warm cache instead of a cold one. - Only answers that can still be served come back (fresh, or within the serve-stale window), and the time spent down counts against them. - A dump taken under different upstreams or routes isn't loaded, so an answer never reaches a client through the wrong upstream group. - The file is owner-only, since it shows what was looked up. It's removed once loaded, so a crash never reloads old data. **In a cluster**, a node that starts with an empty cache (a new resolver pod, a replica after a restart) also asks the other nodes for their most-queried names, this hour and the last, and resolves them in the background (20 a second, A and AAAA), so its first clients find them cached. `[cache] warm_names = 300` sets how many; `0` turns it off. ## Cache tools The **Cache** page (under System) shows each node's cache. In a cluster that's one card per node; Kubernetes pods are told apart by pod name. - **Per node:** the hit rate over the last hour and since start. How many answers it holds, and how much of its memory budget that uses. The last hour's lookups, prefetches, stale answers served, and evictions. Whether it started warm (how many answers it reloaded from the dump, or why it didn't). The `[cache]` settings in effect. - **Charts:** the hit rate and the lookups per 15 s over the last hour, one line per node. A pod that just started shows a low hit rate while its cache fills. - **What's cached:** what each cache holds by kind (answers, names that don't exist, no data, SERVFAIL, stale answers kept for serving stale, DNSSEC-validated), and its top 25 entries by hits, size, or nearest expiry. These walk the whole cache (a few milliseconds per 100,000 entries, one shard at a time, off the query path), so they load when you open the page or press **Refresh**, not every few seconds. Select a name to look it up. - The query log's **Why?** drawer has the same lookup for the name you clicked. Each node samples its cache counters every 15 s and keeps an hour of samples (about 15 KiB). API: `GET /api/v1/cache/stats` (with `settings`, `warmStart`, and `history`) and `GET /api/v1/cache/entries?sort=hits|bytes|expiring&limit=25&node=`, which needs a viewer or the agent scope `analytics:read`. - **Look up** lists every cached answer for exactly that name: the query type (and the variants for clients that set DO or CD), the response code, the number of answers, whether DNSSEC validated it, how long it stays fresh (or how long it has been stale), and its hits. Looking doesn't count as a hit. - **Flush this name** removes it; tick *and everything under it* to flush a whole subtree (`example.com` takes `www.example.com` with it). **Flush everything…** empties the cache after a confirmation. Flushing needs the operator role and is audit-logged as `cache.flush`. - In a cluster, each node has its own cache. Stats and lookups show one row per node, and a flush applies to every node unless you pick one. A node that can't be reached is listed with the error, and the others still flush. - Flushing doesn't clear devices' own caches; a phone or browser may keep the old answer for a few minutes. Blocked answers are never cached, so unblocking needs no flush. To change what a name resolves to, use a local record instead. API: `GET /api/v1/cache/stats`, `GET /api/v1/cache/lookup?name=`, and `POST /api/v1/cache/flush` with `{"name": "example.com", "subtree": true, "node": "pi"}` (every field optional; an empty body empties every node's cache). Agents need `analytics:read` to look and `ops:cache` to flush. ### Is the cache the right size? Each node's card on the Cache page has a **Size** line: whether a bigger cache would answer noticeably more lookups from memory, or a smaller one would do. - **How it's estimated:** the cache follows a fixed sample of names (1 in 16) after they're evicted. When one of them is asked again while its answer would still have been fresh, the number of evictions in between says how much bigger the cache needed to be to still hold it. That gives the extra hits a cache 25%, 50%, and 100% larger would have had (the sample's count times 16). This is the SHARDS technique: a uniform sample of names keeps their reuse distances exact, so the estimate is good with a small fraction of the memory. It runs when an answer is stored or evicted, never on a lookup, and needs about 3 bytes per cached answer. - **The verdict:** *could grow* when a size 25%, 50%, or 100% larger would have answered at least 1% more lookups from memory (the smallest such size is suggested, with the `max_bytes` to set); *could shrink* when, after 100,000 lookups, nothing was ever evicted and the cache never held half its budget (it suggests the peak plus half, at least 4 MiB); *right size* otherwise; *learning* before 10,000 lookups. - Since start: a restart starts the estimate over. In a cluster each node judges its own cache. - `GET /api/v1/cache/stats` has it as `sizing`, the MCP tool `cache_advice` reads it, and `/metrics` has `telltale_cache_ghost_hits_total{size="1.25x"|"1.5x"|"2x"}` (estimated, cumulative) and `telltale_cache_peak_bytes`. ## DNSSEC validation TelltaleDNS can check DNSSEC signatures on forwarded answers itself, instead of trusting the upstream: ```toml [dnssec] mode = "validate" # off (default) | validate | permissive negative_trust_anchors = ["corp.example"] # internal zones that aren't signed ``` - **Signed and valid:** the answer carries AD, for clients that ask (DO or AD set). - **Unsigned:** it's answered as usual, without AD. - **Signed but broken** (forged, expired, or tampered): SERVFAIL with Extended DNS Error 6 (DNSSEC Bogus). In `permissive` mode it's answered anyway and only counted, which is useful for trying validation out. - **NXDOMAIN and "no such record"** answers are checked too, using the zone's NSEC or NSEC3 proofs. - **Opting out:** - a client that sets CD (checking disabled) gets the unvalidated answer; - names under a negative trust anchor, or under a route with `dnssec_nta = true`, aren't validated. Domains forwarded from the UI's "Send a domain to another server" set that automatically. - **Speed:** it works on cache misses only, and a validated answer is cached with its AD bit. The first lookup in a new zone fetches its keys, and a cold start fetches the root's and the TLD's; after that they're cached. - **Counted** in `telltale_dnssec_validation_total{result="secure|insecure|bogus|indeterminate"}`. In `permissive` mode each bogus answer is also logged at INFO with its name, type, and EDE code (`DNSSEC: bogus answer served (permissive mode)`), so a trial shows which names `validate` would refuse. - **Indeterminate** (validation couldn't finish, such as the validator's depth limit): the answer is fetched again without validation and served without AD, in both modes. - **Unsigned zones:** before refusing an answer, TelltaleDNS proves whether its zone is simply unsigned (the parent's signed denial of the zone's DS, which must also show that the name is a delegation: NS present, SOA and DS absent), and if so serves it as insecure. This corrects the validation library's verdict on CNAME chains in unsigned zones (`www.netflix.com`, `www.amazon.com`); proven zones are remembered for 15 minutes (ADR-098). The whole validation, proof included, ends within twice the query budget (4 s by default), at most 8 proofs run at once, and a name that couldn't be proven isn't tried again for a minute: when it can't be proven, the answer stays bogus (or, with no answer at all, stale or SERVFAIL as for any upstream failure). - **Before switching to `validate`:** run `permissive` for a while and read the log; only names that are really broken (like `dnssec-failed.org`) should show up. - **Non-existent names answered locally** (RFC 8198, `aggressive_nsec`, on by default): a validated "no such name" from a zone signed with NSEC proves a whole range of names absent. A later question inside a proven range is answered from it: NXDOMAIN (or NODATA for a missing type), signed, with AD, and without asking upstream. A typical win is a home network's stray queries for made-up top-level names (`printer.lan`, `wpad.home`), which the root zone's NSEC records cover. Only proofs that validated as secure are used, only for as long as their TTL and the zone's negative TTL allow, and never below a delegation. Zones signed with NSEC3 (hashed names) still ask upstream. Counted in `telltale_dnssec_synthesized_total`; `[dnssec] aggressive_nsec = false` turns it off. - **Why it failed:** a bogus answer's SERVFAIL says why when its signatures show it: EDE 7 (signature expired), 8 (signature not yet valid), 10 (signatures missing), else 6. A clock that's far off shows up as 7 or 8 everywhere, which is worth checking first. - **Trust anchors:** the root's 2017 and 2024 keys are built in, so the root key rollover needs nothing from you. To use other anchors (a future key before TelltaleDNS is updated, or a test root), give a file of DNSKEY records in zone-file form, as `dig DNSKEY . +noall +answer` prints them or Unbound's `root.key` holds: ```toml [dnssec] mode = "validate" trust_anchors_file = "/etc/telltale/root.key" ``` It replaces the built-in anchors; a file that can't be read or holds no key is logged and the built-in anchors stay. Automatic tracking of key changes (RFC 5011) isn't done (ADR-086). ## Clusters Nodes can form a cluster: the first node creates it and holds the cluster's certificate authority (it's the **primary**); others join with a token and then keep an encrypted, mutually authenticated link (mTLS over HTTP/2) to it. **The primary's configuration and blocklists reach every node within seconds.** Failover is manual, or automatic with a third voter such as a small witness (below), and **every node's dashboard and query log show the whole cluster**. Changes made in any node's UI or API go to the primary. DNS never depends on the cluster: a node answers the same whether its peers are up or not. **Finding the primary.** A node dials the addresses it joined with and the ones the primary advertises; if none answers, it attaches to another eligible node, so it still gets the cluster's state. While it is attached to a node that isn't the primary, it looks for the primary again every 2 to 30 seconds (jittered, backing off) and moves back as soon as it answers, with `cluster: the primary answers again` in the log. It never moves to an address of a node it knows is not the primary. **One view of the whole cluster.** Open any node's UI and the dashboard, top lists, latency, groups, query log, and live query stream cover every node: - The node you're on asks its peers over the cluster link and merges what they send: - counts add up; - top lists merge their counts and error bounds; - latency percentiles are exact: the nodes send their latency histograms, and the merged histogram gives the percentiles (a node on an older version sends percentiles, which are weighted by its query count instead); - query-log rows interleave by time, and each row shows the node that answered it. - A node that doesn't answer within 2 seconds is left out, and the page says so ("Partial results: pi didn't answer"). It never hangs. The API lists those nodes in `missingNodes`. - `scope` on a stats or query-log request narrows it: `cluster` (the default, every node), `site:` (the nodes of one site, e.g. `site:k8s`), `node:` (one node, by site, pod, or node ID), or `node:local` (the node you're asking). For example `GET /api/v1/stats/summary?scope=site:home-pi`. An unknown site or node is a 400; `missingNodes` lists only nodes inside the scope that didn't answer. - The live query stream follows every node too, about half a second behind for other nodes, each row naming its node. The rate cap is shared between the nodes. `scope=node:local` watches only the node you're on. - Settings and Explain are always this node's own. - Nodes answer each other's reads whether or not their own API is on. **Keeping a node's query log on another node (ship mode).** A node with little or wear-sensitive storage, such as a Pi on an SD card or a pod without a volume, can hand its query log to another node: ```toml [telemetry] mode = "ship" [telemetry.ship] # to = "k8s" # a node ID or site; default: the primary # buffer_bytes = "64MiB" # kept here while the target is unreachable; the oldest goes first # interval_secs = 300 # how often the log is closed and sent ``` - The node writes its query log as usual, but keeps at most `buffer_bytes`. At least every `interval_secs` it closes the current file and sends it to the target over the cluster link. Once the target has checked and stored the whole file, the node deletes its copy. - The target keeps shipped logs under `/qlog-nodes//`, with its own retention. Its query-log search includes them, and each row shows the node it came from. - Every row lives in one place at a time: the sender's buffer until it's delivered, then the target. So cluster-wide searches never count a row twice, and recent rows are searchable from the sender through the cluster until they're delivered. - If the target is down, files wait in the buffer and go out when it's back; beyond `buffer_bytes` the oldest are dropped. Metrics: `telltale_qlog_ship_pending_segments`, `telltale_qlog_ship_errors_total`, and on the target `telltale_qlog_received_segments_total`. - Dashboard counts go too: every `interval_secs` (at most a minute) the node sends its last 30 minutes of per-minute counts to the target, which keeps them for 7 days. While the node answers, the dashboard uses its own numbers; once it's gone (a Kubernetes resolver pod replaced in a rollout), the target's copy keeps its traffic in the cluster's history instead of the totals dropping. Ranges past 7 days use what each node kept itself. - To spare an SD card entirely, put the buffer on a RAM disk: the query log lives in `/qlog`. **What's shared and what stays per node.** The primary shares upstreams, routes, local records, lists, groups, devices, access rules, rate limits, and special names, including what was added in its UI or API. Each node keeps its own `[node]`, `[[listen]]`, `[cluster]`, `[api]`, `[auth]`, `[telemetry]`, and `[cache]` from its own config file. So: - **Change configuration in any node's UI or API.** On a replica, a change to devices, local names, or forwarded domains is passed to the primary: - the primary applies it, and it reaches every node within seconds; - the answer you get is the primary's; - the primary's audit log shows who made it and where, e.g. `alice via pi`, and the replica's audit log shows it too. - `If-Match` uses the primary's configuration version, so it works the same on every node. - If the primary can't be reached, the change is refused with 503 and nothing changes; DNS keeps answering. - Config *files* are different: edit the primary's file. The shared sections of a replica's own file are ignored once it has synced. - **Replicas don't download lists.** The primary compiles them, and replicas fetch only the parts that changed. - **A replica keeps serving what it last received** if the primary is down, including after a restart (it starts answering from `/cluster/applied.json` in milliseconds). - **Every version is signed** by the cluster's key; a replica rejects anything else. - **Users and tokens are the primary's, everywhere.** Users (passwords, roles, two-factor settings, recovery codes) and API and agent tokens are made and changed on the primary, and every node takes them within seconds, so you sign in to any node with the same account and a token works against any node. Only hashes travel, over the cluster's encrypted channel. - On a replica, changing a user, a password, two-factor settings, or a token is refused with a pointer to the primary. - Sessions stay per node: signing in to a second node is a second sign-in (with SSO, a silent redirect). - **Acknowledged anomalies are the primary's, everywhere** too ([Acknowledging findings](#acknowledging-findings)): any node takes an acknowledgement and passes it to the primary, and every node gets the set with the next version. - Users that exist only on a replica (an admin made before it joined) keep working there. If the primary has a user of the same name, the primary's account (and password) takes over on every node. - A promoted replica becomes the place to manage users, with everything it had synced. - **Records for one node only:** `node_only = true` on a `[[record]]` keeps it on the node whose file has it. The primary doesn't share it, and a replica keeps it next to the cluster's records. An example is a name that should only resolve on the Pi: ```toml [[record]] name = "pi.home.arpa" type = "A" value = "192.168.3.2" node_only = true ``` - **Upstreams and routes for one node only:** `node_only = true` works the same way on `[[upstream]]`, `[[upstream_group]]`, and `[[route]]`, for a server only some nodes can reach. The Helm chart marks its route to CoreDNS this way (`kubernetes.enabled`), so a Pi in the same cluster doesn't try to reach `cluster.local` it can't get to. Nothing shared may point at a node-only entry: a shared group can't list a node-only upstream, and a shared route, a client group's `upstreams`, or `upstream_check.reference` can't name a node-only group. If a node's own node-only upstream or group has the same name as a shared one, the node's own is used. - **Leftovers in a replica's file:** if its file still sets shared sections (upstreams, lists, records, groups…), the primary's replace them. The node logs a warning naming them, and the Cluster page's **node_settings** check fails until they're removed. **When the primary is gone: promote another node (manual failover).** - **Who can be promoted:** a node joined with `--eligible` using a token made with `telltale cluster token create --eligible` (a plain token brings a plain member). The primary shares the cluster's signing key with eligible nodes once they connect, so one of them can take over. A node that joined with the Helm chart's bootstrap secret (a resolver pod) is always an ephemeral member: it can't be eligible or a witness, whatever its settings say. Give eligible nodes `--advertise` URLs, so a returning old primary can be reached and told to step down. - **How:** use **Promote this node…** on the Cluster page (admin, live; it asks for your password again, and your authenticator code if you use two-factor sign-in), or stop the node, run `telltale cluster promote`, and start it. Through the API, a signed-in session sends `password` (and `totp`); an API token with admin scope doesn't need them. It's refused while the primary is up, except over an emergency primary: a node that can publish (a Git-managed node, under the `gitops` authority) may take over from it while it runs. - **What happens:** - the node takes a new *epoch* and continues from the last version it applied; - every node follows it; - an old primary that comes back steps down by itself when it sees the newer epoch. - **Changes the old primary made in the meantime aren't applied anywhere.** They're listed under **Conflicts** on that node's Cluster page, with the settings they touched, so you can make them again on the current primary. **Removing a node.** A node that is gone for good stays in the member list as a down node: it counts as a voter in automatic failover and keeps `TelltaleDNSClusterPeerDown` firing. This happens when you rebuild a machine and join it again, because it comes back under a new node ID. The Cluster page lists such a record as a down node (a record that hasn't connected since the node you're looking at started appears after about a minute and a half). Remove it: - **How:** **Remove** on the node's row of the Cluster page (admin), `telltale ctl remove-node `, or `DELETE /api/v1/cluster/nodes/`. Use the node ID the Cluster page shows, not the site name. Add `--dry-run` (`?dryRun=true`) to see what would happen first. It works from any node (a replica sends it to the primary) and under a Git config authority, since the member list isn't configuration. - **A node that is up** is refused unless you add `--force` (`?force=true`; the page asks you to confirm). Its connection is closed, and every node refuses it from then on, even though its certificate is still valid for up to 90 days. Stop it and clear its cluster directory before it joins again with a new token. - **The primary can't be removed.** Promote another node first. - **Voters:** a removed node stops voting. Automatic failover needs three voters, so removing one of three turns it into manual until another eligible node or a witness joins; the answer says so. - **The cluster key:** an eligible node holds a copy of it. Removing the node stops the cluster accepting it, but if the machine is lost or can't be trusted, also run `telltale cluster rotate-ca`. For a record whose machine is gone, there is nothing more to do. **Upgrading a cluster.** Upgrade one node at a time, **replicas first, then the primary**. DNS keeps answering throughout if your clients have two DNS servers (each node is one). - Versions one cluster protocol apart work together (N and N−1, either way round), so a half-upgraded cluster is fine. - The Cluster page's **versions** check lists the mix until the last node is upgraded. - A replica older than its primary keeps working, unless the primary starts using settings the older version doesn't know. Then the replica keeps serving its last version, and its sync status says to upgrade it. **Certificates renew themselves.** Each node's cluster certificate lasts 90 days, and a node renews it when fewer than 30 days remain (checked every 6 hours). It keeps its key and node ID. Nodes holding the cluster key issue their own; others ask the primary over the cluster link, which signs only for the node asking. A renewal that fails (for example, with the primary down) is retried every 10 minutes, and shows in the Cluster page's events and in `telltale_cluster_cert_expiry_timestamp_seconds`. The certificate check on the Cluster page and the Helm chart's alert warn below 14 days. **Rotating the cluster CA.** The cluster CA lasts 10 years. To replace it (a suspected leak of a node holding the cluster key, or policy), run on the primary while it runs: ```sh telltale cluster rotate-ca # start telltale cluster rotate-ca --status # phase, and which members it waits for ``` The primary drives three steps, each only when every member is ready, so no link breaks and DNS keeps answering: 1. Every node trusts the new CA next to the old one (sent in the signed configuration). Eligible nodes also get the new key. 2. The new CA signs. Every node renews its certificate from it within a minute, keeping its key and node ID. 3. Every certificate is from the new CA, so the old one is retired and its key is deleted. A member that's away holds the rotation until it's back; the Cluster page shows the rotation and whom it waits for. Ephemeral members (resolver pods) don't hold it up. A node that's away for the whole rotation can't connect afterwards and must join again with a token. Join tokens made before a rotation pin the old CA, so make new ones afterwards. Rotation helps when the cluster key may have leaked but hasn't been used. Whoever holds the current key can sign configuration and rotate trust to a CA of their own, exactly as the primary does. If that may already have happened (configuration or a rotation you didn't make), re-create the cluster: `telltale cluster init` on one node, and join the others with new tokens. **Automatic failover (with a witness or three eligible nodes).** Two nodes can't tell "the other node is down" from "the link between us is down", so automatic failover needs a third vote: another eligible node, or a **witness**, a tiny vote-only process that can run on a NAS, router, or small VM. ```sh # On the witness host (a config with [node] data_dir and [cluster] listen is enough): # The token from `telltale cluster token create --witness` on the primary. telltale cluster join --witness --advertise https://witness.lan:8443 -c witness.toml telltale cluster witness -c witness.toml # run it as a service # On the primary, then restart it: telltale cluster set-failover auto ``` - **How it decides:** every eligible node and witness is a voter. The primary renews a 15 s *lease* with a majority every 5 s, and publishes only while the lease holds. If it disappears, the eligible nodes wait for the lease to run out, and one is elected by a majority vote in a new epoch, typically within 15–25 s. - **Never two primaries writing:** a voter votes once per epoch and won't vote for anyone else while the lease it granted runs. A primary that loses its majority (for example, cut off by a network split) stops taking changes before anyone else can be elected. Proven by a simulator that runs 10,000 random partition, crash, and clock-drift schedules on every CI run. - **Clock changes don't matter.** Leases are timed on a clock that never jumps, so a Pi (no battery-backed clock) setting its time from NTP after boot, or a machine resuming from sleep, can't cut a lease short. A voter that restarts keeps any lease it granted for a full 15 s. - **DNS never waits for any of this.** Every node keeps answering from its last configuration; only configuration changes pause (503) until a primary with a lease exists. - **Seeing it:** the Cluster page's **Failover** line (voters reachable, the lease, the latest vote) and the `failover` check. Metrics: `telltale_cluster_primary`, `telltale_cluster_epoch`, `telltale_cluster_failover_auto`, `telltale_cluster_lease_held`. - **Manual promotion is refused while automatic failover runs**; `telltale cluster set-failover manual` on the primary switches back. With fewer than three voters, `auto` acts as `manual`, and the Cluster page says so. - **Under a Git config authority,** an elected node that isn't Git-managed becomes an emergency primary: it keeps the cluster coordinated on the last version. While a Git-managed node is up, the other nodes leave the election to it (they still vote), so the emergency primary exists only while no Git-managed node can be primary. When one is back, it takes the role back through the election: it asks the emergency primary to hand over (see *Where configuration comes from*, below). **Configuration from a Git repository.** The cluster can take its shared settings (upstreams, lists, groups, devices, records…) from one file in a Git repository. The primary fetches it, checks it, and hands it to every node, so all nodes serve the same commit: ```toml # On every node that may become primary: [cluster.git] repo = "https://github.com/me/homelab" ref = "main" # branch, tag, or commit path = "telltale/shared.toml" # the shared settings, in TelltaleDNS's format # credentials_file = "/etc/telltale/git-token" # private repositories: a token, or user:password # poll_secs = 60 # webhook_secret_file = "/etc/telltale/git-hook" # then add a push webhook to /api/v1/hooks/git # require_signed = true # only SSH-signed commits by... # allowed_signers_file = "/etc/telltale/allowed_signers" # ...these keys (git's format) # allow_rewind = false # refuse force-pushes and rewinds ``` - **Safe by default:** a commit is used only if its file loads and validates (merged with each node's own settings), and it descends from the commit in use. Anything else is refused: the cluster stays on the last good commit, and the Cluster page, the event log, and the `TelltaleDNSGitCommitRefused` alert say why. - **Signed commits:** with `require_signed`, only commits SSH-signed by a key in `allowed_signers_file` are accepted (`git config gpg.format ssh`; Ed25519 keys). - **Webhook:** point the repository's push webhook at `https:///api/v1/hooks/git` with the secret from `webhook_secret_file`. Changes then apply within seconds, not at the next poll. - **Outages don't matter to DNS:** with the repository unreachable, every node keeps serving the last commit (`telltale_cluster_git_failing` and an alert after 15 minutes). - **What you see:** the Cluster page shows the repository, the commit in use (author, message, signer), and each node's commit. `telltale_cluster_config_commit{commit=…}` on every node should be the same. - **Changes go through Git:** with a Git source the cluster is GitOps-managed, so the UI and API don't change shared settings (409 `gitops_managed`). - **Failover:** any eligible node with the same `[cluster.git]` can take over and keep following the repository. A node configured with a different repository, ref, or path refuses to fetch, since the cluster is pinned to its source. - No `git` program is needed: TelltaleDNS speaks Git's protocol over HTTPS and fetches only that one file. **Where configuration comes from (config authority).** - **Choosing it:** `telltale cluster init --config-authority gitops` (or `telltale cluster set-authority gitops` on the primary, then restart) says the cluster's configuration comes from Git. - **What it changes:** - only nodes whose own configuration is Git-managed (`[cluster] config_source = "gitops"`, set for the Kubernetes node by its Helm values) may then publish it; - configuration changes through the API or UI are refused on every node (409 `gitops_managed`); - another node can only be promoted as an *emergency* primary, which keeps the cluster coordinated on the last version and publishes nothing new until a Git-managed node is back. When one is back, it takes over by itself after a minute (an eligible, Git-managed node that sees the emergency primary up), and the emergency primary steps down. With `failover = manual` you can also promote it right away from its Cluster page or with `telltale cluster promote`. The default is `api`: the primary's own file and UI. - **Taking over with automatic failover.** With `failover = auto` a role can't be set by hand (the votes decide, and a promoted node would hold no lease, so it couldn't publish). The Git-managed node takes over through the election instead: after seeing the emergency primary up for a minute, it asks that node to hand over. The emergency primary stops renewing its lease, keeps answering DNS, and the Git-managed node is elected within the usual 15 to 25 seconds (configuration changes wait for that, DNS doesn't). Nodes that aren't Git-managed don't stand in that election. If the Git-managed node can't win (for example it hasn't received the cluster key yet), the emergency primary keeps the role after a minute and is asked again later; the Cluster page's event list shows `takeover` and `handover` events. Only an eligible, Git-managed node that is up can ask. - **Which failover mode?** With the `gitops` authority only a Git-managed node can publish, so `manual` failover is enough for many clusters: replicas keep serving the last version while that node is away, and you promote an emergency primary only if you need the cluster coordinated for a long outage (certificate renewal waits for a primary). `auto` does that without anyone present, and the Git-managed node takes the role back by itself. Run these as the user telltale runs as (`sudo -u telltale` for native installs, `docker compose exec telltale` for Compose), then restart telltale. ```sh # On the first node: create the cluster. List every URL other nodes might use to reach it. telltale cluster init --name home --advertise https://192.168.3.2:8443 --site home-pi # Still on the first node: a join token (reusable until it expires; treat it like a password). telltale cluster token create --ttl 1h # brings a plain member (follows, can't be promoted) telltale cluster token create --ttl 1h --eligible # may bring a node that can be promoted telltale cluster token list # the tokens still valid: their IDs and expiry telltale cluster token revoke 3f9c2a1b7d0e # stop one early (nodes that joined stay members) # On the joining node: join, then restart it. telltale cluster join tt_join_... --site k8s # On any node: this node's identity. telltale cluster status ``` - **The cluster port** is `[cluster] listen` (default `0.0.0.0:8443`). It opens only once the node is in a cluster. Only the first node's port must be reachable: joining nodes connect out to it, so they can sit behind NAT. If you also serve DoH on 8443, change one of them (`telltale config check` warns about the clash). - **Trust:** the token carries the fingerprint of the cluster's CA, so a joining node can't be tricked into joining another server; the token's secret proves the node may join. Each node gets a certificate valid for 90 days. - **State** lives in `/cluster/` (keys are readable only by their owner). Removing that directory takes a node out of the cluster. - **The Cluster page** (and `GET /api/v1/cluster`) answers "is the cluster healthy and serving?" on any node. It shows: - every node's role and site, whether it's up, its link and round-trip time; - each node's configuration version and how far and for how long it's behind; - whether each node is serving DNS (ready, queries per second, SERVFAIL share, upstream p90), its version, uptime, and certificate expiry; - six checks (peers up, primary present, in sync, no sync errors, serving, certificate) with what to do when one fails; - a timeline of joins, connections, published and applied versions, and failures. The cluster's summary stays at the top; the rest is in tabs: **Nodes** (the topology and the node table, with Maintenance…), **Health** (checks, conflicts, machines), **Rollouts** (staged rollouts and versions), and **Events**. A tab with something wrong says so (failing checks and conflicts, a version baking or a pin, failures in the last hour), and a pin shows on every tab. The tab is in the address (`#/cluster?tab=health`), so it can be linked. The page refreshes every 5 seconds. - **Metrics:** - `telltale_cluster_peers{state}`; - per peer: `telltale_cluster_peer_up`, `telltale_cluster_peer_rtt_seconds` and `telltale_cluster_peer_config_lag`; - this node: `telltale_cluster_config_seq`, `telltale_cluster_behind_seconds`, `telltale_cluster_sync_error` and `telltale_cluster_cert_expiry_timestamp_seconds`. The Helm chart's PrometheusRule alerts on a peer down for over a minute, configuration behind for over 30 s, failing syncs, and a certificate within 14 days of expiry. ### Compare nodes **Cluster → Compare** puts every node side by side over the same window (1 hour to 30 days), for when "something seems odd with that node": - One row per measure: queries per second, the SERVFAIL share, the slow share (answers over the latency objective's threshold), the share answered from the cache, the share of upstream exchanges that failed, p50 and p99. Each cell has the window's number and a sparkline; the sparklines of a row share one scale. - **A node that deviates is outlined**, with one line saying how and since when. With three or more nodes, a node is odd when it sits at least 3 robust standard deviations (from the median and MAD of the *other* nodes, with a floor) away from them and its own 95 % interval leaves their median out. With two nodes, it's odd when the two 95 % intervals don't overlap by a clear margin, and the worse one is marked. **Since** is when its excess over the others began. - **A grey note** says when a node only looks worse: the same number of failures as the others on a much smaller base, such as a Pi with a tenth of the traffic. - **Table** shows the same numbers with their 95 % intervals, such as `0.04 % (0.02–0.06 %)`: a wide interval means few answers. **Open in dashboard** opens the dashboard for that node alone. On a phone, pick one measure at a time. - The tab says **1 odd** from any tab of the Cluster page. It's a lens, not an alarm: it doesn't change the health icon or alert. Nodes that don't answer are listed and the rest compared; one node alone has nothing to compare. - `GET /api/v1/cluster/compare?from=-24h` and the MCP tool `compare_nodes` return the numbers and the findings. Grafana's **Nodes** row has the SERVFAIL and slow shares by node and a table that highlights a node with little traffic. ## Taking a node down for maintenance Swapping a Pi's SD card, moving a box, or upgrading a node by hand: put the node in maintenance first. For a bounded time (an hour by default, a day at most) it: - **reports not ready:** `/readyz` answers 503 with `{"ready":false,"reason":"maintenance"}`, so load balancers and Kubernetes stop sending it new queries; - **keeps answering** every query that still arrives (a router's secondary DNS entry, a device with the address typed in): no listener closes; - **is left out** of alerts (`node_down`, `sync_lag`, `probe_failing`, `cert_expiring`, `disk_full`, and its share of `servfail_rate`), the health level, and the Cluster page's checks; an alert about it that was firing resolves with "(maintenance)"; - **isn't elected** primary (it still votes); - **stays in maintenance across a restart:** the window is in `/maintenance.json`, so a node rebooted inside it comes back not ready until the window ends or someone ends it. Everything else carries on: the cache, lists, replication, the query log, probes (they keep measuring), the API and UI. The Cluster page, `cluster_status`, and the health panel show the node, its window, and its real figures the whole time. ```sh telltale ctl maintenance start --for 30m --reason "SD card swap" # this node telltale ctl maintenance start --for 2h --reason "kernel upgrade" --node pi # another node telltale ctl maintenance status telltale ctl maintenance end --node pi ``` Or on the Cluster page: a node's **Maintenance…** (how long, why) and **End maintenance**. Over the API it's `POST /api/v1/nodes/{id}/maintenance` with `{"forSecs": 1800, "reason": "…"}` and `DELETE` to end it (`id` is `local`, or a node's ID, site, or pod name). Maintenance is the node's own state, not configuration: any node passes the request to the node it's for (a node that's down answers 503 `node_unreachable`), it works in a GitOps-managed cluster, and it's audit-logged as `node.maintenance.start` and `.end` on that node. AI agents use `start_maintenance` and `end_maintenance` (`ops:maintenance`, at most 2 hours). Starting again replaces the window. `[node] maintenance_default_secs` sets the default length. **The primary.** Under automatic failover with another eligible node online, maintenance on the primary hands the role over first: the primary stops renewing its lease, another node is elected within the lease window (configuration changes pause for about 30 s, as in any failover), and the old primary follows for the rest of the window. The form shows this as a checked box ("hand the primary role to another node first"; `handover: false` over the API keeps the role). In manual failover, or with no other eligible node online, the primary stays primary and keeps publishing; the answer says so. Promote another node first if you're taking it offline. **Kubernetes.** One readiness probe serves every Service that selects a pod. A resolver pod in maintenance leaves the DNS Service, which is the point. The controller pod in maintenance also leaves the API Service: use another node's UI or `kubectl port-forward`. A StatefulSet rollout waits for readiness, so a pod in maintenance holds a rollout until its window ends. The chart's startup probe asks `/readyz?startup=1`, which leaves maintenance out, so a pod restarted inside its window isn't killed for it. For routine controller work, `kubectl drain` and rollouts remain the usual path; maintenance mode is for the Pi and individual pods. **Docker Compose.** The healthcheck uses `/readyz`, so the container shows as unhealthy during maintenance (Compose doesn't restart it for that). **Alerts and metrics.** An alert rule with `when = "maintenance"` tells a destination when a node enters maintenance and when it leaves (not in the defaults). `/metrics` has `telltale_node_maintenance` (0/1), `telltale_node_maintenance_until_seconds`, and `telltale_node_maintenance_total`, plus `telltale_cluster_peer_maintenance_until_seconds{node,site}` for peers. The Helm chart's node alerts leave out a node whose last reported window is still open (also when it stopped reporting), and the Grafana dashboard has a **Maintenance** timeline. A node that's really failing during its window stays quiet until the window ends: keep windows short. ## Failover drills Your router hands devices two DNS servers, say the Pi and a second machine. On a healthy network a device only ever uses its first one: most systems switch to the second only when the first stops answering. So nothing shows whether the second one works for every device, until the day the Pi's SD card dies. A **drill** finds out on purpose, for a short time. For 15 seconds to 5 minutes (60 by default), one node stops answering the devices you choose (some devices, a group, or everyone) while the other nodes keep serving: - **UDP queries get no answer** and TCP and DoT connections are closed without one: to a device, the node looks dead. DoH and DoQ queries get no answer (their connections stay open, since other queries share them). Each such query is logged with the status **drilled**. - **Never left unanswered:** the node itself (`127.0.0.1`, `::1`: its probes and local programs), its infrastructure (the default gateways, `TELLTALE_NODE_IPS`, `[clients] infrastructure`), the cluster's nodes, and [excluded](#keeping-names-and-devices-out-of-the-log) devices. - **It's a maintenance window too:** the node reports not ready, its alerts pause, and a primary hands its role over first (as for [maintenance](#taking-a-node-down-for-maintenance)). Unlike maintenance, **a restart cancels a drill**: the node comes back answering everyone. Every node keeps a record of who asked it while the drill runs, including nodes the drilled one has no direct connection to (a resolver pod only talks to the controller, and a node outside Kubernetes to no resolver pod): the primary passes the notice and the request for the records on to them. A minute after the drill ends, the drilled node gathers them into a **report**: | Verdict | Meaning | |---|---| | **followed** | another node answered it after its first unanswered query, before the drill ended or within 5 seconds after: its second server works. With the time it took (`timeToFailoverMs`). | | **stuck** | 3 or more unanswered queries and no such answer: it has no working second server, or its second server is the same node. | | **quiet** | nothing unanswered, or too little to tell (it may have been idle, or asked only the other node). An answer elsewhere after the drill is over proves nothing: the drilled node answers again by then. | | **elsewhere** | not targeted, but it asked another node during the drill: for information. | **Coverage** is followed / (followed + stuck). Fifteen minutes after the drill, the report also says whether each device that followed **came back** to the drilled node or **stayed** on the other one (Windows stays for about 15 minutes; that's normal). Start small: one device of your own for 30 seconds, then a group, then everyone. Devices without a working second server lose DNS for the drill's length. **REFUSED instead of silence.** With `"mode": "refused"` (the form's "REFUSED at once"), the node answers its targets REFUSED (extended error 14, "failover drill") instead of nothing. It's faster: no device waits out a timeout. It's also less realistic: some devices try their next server on REFUSED, others just fail the lookup, so a device that is `stuck` here may still fail over from a dead server. Use the default for the real question. **Every month, at night.** A `[drill]` section schedules drills: ```toml [drill] schedule = "monthly" # "off" (the default) nodes = ["home-pi"] # the nodes that drill themselves: sites, pod names, or node IDs for_secs = 30 # 15 to 300 hour = 3 # local hour it starts in tz = "Europe/Berlin" # default: the system's zone # group = "iot" # or clients = ["192.168.1.20"]; neither: everyone # mode = "refused" # default "drop" ``` Each named node drills itself once a month, during that hour, on a day in the first week picked from its name and the month (so the day moves, and two nodes rarely share it). It tries once a minute through the hour; if a precondition fails all hour, it skips the month. It never drills within 7 days of another drill on the same node. To get the report, add an alert rule with `when = "drill_report"` (it covers drills started by hand too), for example to an email destination. ```sh # on the Cluster page: a node's Drill… (how long, whom, why), Preview, then Start drill curl -X POST https://pi:8443/api/v1/nodes/pi/drill?dryRun=true -d '{"forSecs":30,"targets":{"clients":["192.168.1.20"]},"reason":"does the second server work?"}' ``` A drill is refused (409 `drill_refused`, with the reason) when no other node is ready and answering (a drill would be an outage), the node is in maintenance, a drill ran in the cluster in the last 10 minutes, a rollout is baking or the cluster is pinned, or health is severe. Unknown targets, or targets no device seen on the node in the last hour matches, are 422 `drill_targets`. `?dryRun=true` runs every check and lists the devices without dropping anything. `DELETE /api/v1/nodes/{id}/drill` ends it early (the report still comes); the **Drills** tab on the Cluster page lists every node's reports (`GET /api/v1/drills`, `GET /api/v1/drills/{id}`). Drills are audited as `drill.start` and `drill.end`, leave `drill.started` and `drill.ended` markers on the charts, and need an operator; AI agents plan one with `plan_failover_drill` (the `ops:drill` scope, which no token has by default) and read reports with `drill_report`. At privacy level 2 and above, reports have counts only. **Kubernetes.** A drill on a resolver pod behind one balancer address tests the balancer, not the devices: they never see a server change. The pod reports not ready, the Service stops sending it queries, and the report's `balancerSwitchMs` says how long that took: about 4 seconds with the chart's readiness probe (every 2 seconds, 2 failures). For two real servers (the Pi and the cluster's address), drill the Pi. **Metrics.** `telltale_drill_active`, `telltale_drill_dropped_total`, `telltale_drill_last_coverage_percent`, `telltale_drill_last_stuck`, `telltale_drill_last_failover_ms{quantile}`, and `telltale_drills_total{result}`. A drill never changes the health level. An alert rule with `when = "maintenance"` hears about drills too (the reason starts with "failover drill"). **Which servers devices use.** In the Advanced view, the Clients page has a **Servers** column: each device's queries per node over the last week, and its share on the node it uses most ("mostly pi", "both"; `GET /api/v1/clients/affinity?window=7d`). A device that has only used one server may still have a second one configured; most systems use their first server until it stops answering. Run a drill to find out. ## Rolling a change out in stages A bad change (a typo in an upstream, a list that blocks the bank) normally reaches every node within seconds. With **canary nodes** set, a change reaches them (and the primary) first, and every other node only after a **bake** of 5 minutes, and only if nothing went wrong meanwhile. If something did, the cluster is **pinned** to the version before the change (next section) and you're told why, while the other nodes never saw the bad version. **Off until you set it.** `[cluster.rollout] canaries` is empty by default: changes reach every node at once, as before. Name canaries in **Settings → Cluster → Staged rollouts** (a picker with the cluster's nodes, each site, and "resolver pods"), in the config file, or in the Helm chart's `cluster.rollout`: ```toml [cluster.rollout] canaries = ["ephemeral"] # node IDs, "site:", or "ephemeral" (the Kubernetes resolver pods) bake_secs = 300 # at least 5 servfail_pct = 5 # fail when the canaries' SERVFAIL share is over max(before + 2 points, this) require_traffic = false # true: keep baking until the canaries answered 50 queries, then judge max_bake_secs = 3600 # the longest bake (with require_traffic, or waiting for a canary to apply) fail_on_disconnect = true # a canary gone for a minute after applying fails the change history = 20 # versions the primary keeps for pinning back to ``` `[cluster.rollout]` is shared, so every node knows the canaries. A change to these settings is published at once and never staged itself, so a bad canary list can always be fixed. **The primary runs a change at once.** It's where the change was made, and it serves the version it publishes, so it's judged with the canaries. During the bake the Cluster page says "version 125 runs on k8s-resolver-0, k8s-resolver-1 and the primary; the others stay on 124". **The guard.** Every 5 seconds the primary looks at what each canary reports in its heartbeats. These fail a version at once: - a canary that can't apply it; - a canary not ready for 15 seconds after applying it; - a canary whose listener checks fail after applying it; - a canary gone for a minute after applying it (with `fail_on_disconnect`). The SERVFAIL share of the canaries and the primary during the bake is compared with their share in the same length of time before it, as soon as there are 50 answers: the version fails when it's over `servfail_pct` and over the earlier share by more than 2 points, so a bad change is usually pinned back seconds after it went out, before the bake ends. A version passes at the end of the bake. With fewer than 50 answers by then there's nothing to compare: it passes (only the checks above apply), unless `require_traffic` keeps it baking until there are 50, or until `max_bake_secs`, when it fails. Latency isn't judged: a Pi and a Kubernetes pod differ too much for "slower than before" to mean anything. A pass gives the version to every node, as a new version number with the same content. A fail pins the cluster to the version before it, fires the `rollout_failed` alert, and records the reason and the readings with the version. **Promote or abort by hand.** On the Cluster page while a version bakes: **Promote now** (everyone gets it without waiting) or **Abort** (pin to the version before it; the primary and the canaries ran the change, so they go back too). Both need an admin. **Good to know:** - **A newer change during a bake** replaces the version baking: the canaries get the newer one, and the bake starts over. - **Users, tokens, and the member list** wait for the bake to end before they're published (a version for everyone would carry the canary's configuration). - **No canary online:** a change waits up to a minute after the primary starts (a canary may still be reconnecting), then goes to every node at once ("rollout skipped: no canary online" in the log and on the Cluster page). Canaries set but none online for 10 minutes is the health condition `rollout_stuck`: changes are reaching every node without a bake. - **A restart of the primary during a bake** starts the rollout over. - **Older nodes** (from before this release) are never canaries, even when their site is named: they get each change after the bake. - **Kubernetes** (`mode: scaled`): with `canaries: [ephemeral]` the resolver pods take each change at once with the controller, under the guard's watch, and nodes outside Kubernetes (a Pi that joined) wait out the bake. A pod rescheduled during a bake counts as a canary gone; unpin to try again. ```sh telltale ctl rollout status # the version baking, the canaries and what each runs, the time left, the guard telltale ctl rollout promote telltale ctl rollout abort ``` Over the API: `GET /api/v1/cluster/rollout`, `POST /api/v1/cluster/rollout/promote` and `/abort` (admin; audited as `rollout.promote` and `rollout.abort`); the settings are `PUT`/`DELETE /api/v1/cluster/rollout-settings/default`. AI agents read `rollout_status`. **Metrics** (the primary reports them): `telltale_cluster_rollout_stage` (1 while a version bakes), `telltale_cluster_rollout_info{version,started}`, and `telltale_cluster_rollouts_total{outcome}` (`started`, `promoted`, `failed`, `aborted`, `skipped`, `pinned`, `unpinned`). The Helm chart's PrometheusRule has `TelltaleDNSRolloutFailed`. ## Pinning the cluster to a version A pin makes every node serve an older version again: its shared configuration and its filter snapshot. The guard pins by itself when a rollout fails; an admin can pin any kept version. - **A pin is a new version.** Pinning to version 124 publishes, say, version 131 with 124's content (version numbers never go backwards), and the primary serves it too. - **While pinned, changes wait.** Configuration changes (the UI, the API, agents' plans) answer 409 `cluster_pinned`, with a `diff` of what the pin holds back (JSON Patch, and in the hint one sentence per section). A dry run still works. A Git source keeps checking the repository but publishes nothing; list updates still download, but no new filter snapshot goes out. - **It's loud.** A banner on the Cluster page (pinned to what, since when, by whom, why, and what it holds back), the health condition `cluster_pinned`, and the alert `cluster_pinned` (a rule with `when = "cluster_pinned"`; the Helm chart's `TelltaleDNSClusterPinned` fires after an hour). - **It survives failover.** A replica promoted to primary keeps the pin, and keeps serving the pinned version until someone unpins it there. - **Unpin** publishes the current configuration again, through a staged rollout when canaries are set. A pin doesn't rewrite your files, the entries made in the UI, or Git: undo the bad change there (the diff says what it was), or right after unpinning, while the new version bakes on the canaries. On the Cluster page: the **Versions** table lists the kept versions (when, what made each, its outcome and the guard's readings), with **Pin…** on each (a reason, then a preview of what changes on every node). The pin banner has **Unpin**. ```sh telltale ctl versions # newest first; * = what every node runs telltale ctl pin 3.124 --reason "the new list blocks the bank" --dry-run telltale ctl pin 3.124 --reason "the new list blocks the bank" telltale ctl unpin ``` Over the API: `GET /api/v1/cluster/versions`, `POST /api/v1/cluster/versions/{epoch}.{seq}/pin` with `{"reason": "…"}` (`?dryRun=true` shows the diff), and `DELETE /api/v1/cluster/pin` (admin; audited as `cluster.pin` and `cluster.unpin`). Problems: `version_unknown` (404), `version_blobs_missing` (409, its files are no longer kept), `no_rollout` (409, promote or abort with nothing baking). AI agents use `plan_pin_version` and `plan_unpin` (`cluster:admin`), only where an operator approves each plan (`[agents] require_approval = true`), as for promotion. **What's kept.** The primary keeps the last `history` versions (20) in `/cluster/versions.json`, and their files in its blob store: a configuration is a few kilobytes, and filter files that didn't change between versions are stored once. A node promoted to primary starts its own history (plus the version it was running), so versions only the old primary kept can't be pinned there. **Metrics:** `telltale_cluster_pinned` (0/1) and `telltale_cluster_pinned_to{version}`. ## Monitoring **The dashboard across restarts and upgrades.** - Charts over time come from `/rollups.db`, kept by minute (7 days), hour (400 days) and day. That includes traffic by group, which is stored by group name, so renaming or reordering groups doesn't mislabel history. Minutes saved before this release have no group breakdown. - The current and previous hour's top lists and latency are rebuilt at start from the query log, in the background within seconds. This needs the query log on, kept locally, at privacy level 0. Otherwise they start empty, and completed hours still come from the rollups. - Prometheus counters restart from zero, as Prometheus expects. An HTTP listener (default `0.0.0.0:9153`, set with `[telemetry.metrics] listen`) serves: | Path | Use | |---|---| | `/metrics` | Prometheus scrape | | `/livez` | the process is alive (Kubernetes liveness) | | `/healthz` | the process is healthy | | `/readyz` | 200 once every DNS listener is bound, 503 while starting, shutting down, or in [maintenance](#taking-a-node-down-for-maintenance), with the reason as JSON (Kubernetes readiness); `?startup=1` leaves maintenance out (the startup probe) | This listener needs no sign-in and answers only clients inside `[access] allowed_networks`. The same `/metrics` is on the API port for signed-in users (a viewer token, or HTTP Basic for scrapers that can't send one). Main metrics: | Metric | What it tells you | |---|---| | `telltale_queries_total{proto,status}` | queries by outcome: `cached`, `refreshed` (an expired answer given at once while it's refreshed), `forwarded`, `stale`, `local`, `special`, `blocked`, `refused`, `rate_limited`, `malformed`, `servfail`, `dropped` | | `telltale_query_duration_seconds{path}` | latency histogram per path (`cache`, `upstream`, `local`, `synthesized`) | | `telltale_stage_duration_seconds{stage}` | time spent waiting for upstreams, per query that waited (`stage="upstream"`) | | `telltale_responses_total{rcode}`, `telltale_queries_by_qtype_total{qtype}` | answers by RCODE; queries by type | | `telltale_cache_*` | hits, misses, stale answers served, prefetches, entries, bytes, evictions | | `telltale_upstream_requests_total{upstream,outcome}` | attempts per upstream, `outcome` = `success` or `failure` (timeouts, errors, SERVFAIL/REFUSED) | | `telltale_upstream_failures_total{upstream,kind}` | the failures by kind: `timeout`, `network` (connection, TLS, socket), `bad_response` (mismatched or malformed), `unresolved` (the upstream's hostname), `servfail`, `refused`, `other_rcode` | | `telltale_upstream_duration_seconds{upstream,protocol}` | exchange-time histogram per upstream (p50/p95/p99 in Grafana) | | `telltale_upstream_breaker_state`, `telltale_upstream_latency_ewma_seconds` | circuit breaker (0 closed = healthy, 1 half-open = probing, 2 open = benched; the web UI says Healthy, Probing, Benched); smoothed latency | | `telltale_upstream_connections_opened_total{upstream,outcome}`, `telltale_upstream_connect_seconds_total{upstream}` | connections opened to TCP, DoT, and DoH upstreams (`ok` or `failed`), and the time spent opening them: their ratio is the mean connect time, TCP and TLS together. Many opens per minute mean the pool keeps closing them (see below) | | `telltale_upstream_tls_handshakes_total{upstream,kind}` | TLS handshakes, `full` or `resumed` (from a stored session, a round trip shorter) | | `telltale_upstream_connections_closed_total{upstream,reason}`, `telltale_upstream_connections{upstream}` | pooled TCP and DoT connections ended: `idle` (unused for `idle_timeout_ms`), `server` (closed or reset by the upstream), `write_failed`, `unanswered` (queries waited 3 s with nothing at all coming back: a connection dropped silently along the way; its queries retry at once on a new one), `unused`; and how many are open | | `telltale_upstream_connection_retries_total{upstream}` | queries sent again on a new connection because the pooled one closed just before answering | | `telltale_blocked_total{group,list}` | blocks by the client's group and the deciding list (`list` is `quick`, `schedule`, or `answer_ip` when a quick rule, a schedule, or rebinding protection decided instead) | | `telltale_client_queries_total{client}` | queries per client: off by default; turn on with `[telemetry.metrics] per_client = true` (at most `per_client_cap` clients, default 100; the rest are `client="other"`) | | `telltale_list_entries{list}`, `telltale_list_regex_skipped{list}`, `telltale_filter_*` | list sizes, regex rules a list ships that the snapshot leaves out (over `max_regexes`), snapshot version, rule count, compile time, and `telltale_filter_lookup_mode{mode="indexed"|"walk"}` (1 for the lookup in use; `walk` is the slower fallback when a snapshot has more distinct list combinations than the index holds, 65,536, which is also logged as a warning) | | `telltale_telemetry_dropped_total`, `telltale_qlog_*`, `telltale_ratelimited_total` | analytics that fell behind (answers never wait), query-log writes, rate limiting | | `telltale_udp_*`, `telltale_tcp_*` | listener counters | | `telltale_resident_memory_bytes`, `telltale_uptime_seconds`, `telltale_build_info` | process | | `telltale_config_seq`, `telltale_annotations_total{kind}`, `telltale_annotation_write_failures_total` | the configuration sequence this node applied; [change markers](#change-markers) written, and those that couldn't be | | `telltale_process_starts_total`, `telltale_process_unclean_starts_total` | starts on this data directory, and those after a run that didn't stop cleanly (a crash, an OOM kill, a power cut); kept in `/runs.json`, so they count across restarts (a new pod with an empty volume starts from 1). Settings → System shows them too. | | `telltale_host_*`, `telltale_cgroup_*`, `telltale_data_*`, `telltale_open_fds` | the machine it runs on: see [Host resources](#host-resources) | Counters are kept per thread and summed on scrape, so recording never slows a query or allocates memory. The per-upstream, per-list, and per-client series are built by the analytics thread from query events, off the query path. ### Traffic map **Monitor → Traffic map** answers "what is the network reaching right now?" in one picture, which suits a wall screen: - Each tile is a service, a site, or a name, sized by how often it was looked up this hour (or the last). Blue tiles were answered, red ones were blocked; a stronger colour means busier. - **Services** folds names into the services of the [blocked-services](#blocked-services) catalog (all of YouTube's names make one YouTube tile) and shows other names by site. **Sites** folds each name into its base domain (`rr3.googlevideo.com` → `googlevideo.com`). **Names** shows them as devices asked. - It refreshes every 5 seconds. The five tiles that grew most since the previous refresh get an amber ring and a `+N`. - Point at a tile for the names behind it; select it to open them in the query log. - It draws the busiest 100 names (less their blocked queries) and the busiest 100 blocked names, every node of a cluster, from the same approximate top lists as the dashboard, so it adds no work to answering queries. Each hour starts every count again, so the map looks sparse just after the hour. - Uptime monitors, phones checking their connection, and other background lookups can crowd it: [keep them out](#keeping-names-and-devices-out-of-the-log). ### Where does time go? The dashboard's **Where time goes** card (Advanced view) has two views: - **Percentiles:** this hour's p50, p90, p99, and max per answer path and transport, and the time spent waiting for upstreams. - **Heatmap:** answers over time (minutes for 1 to 24 hours, hours for 7 days) in the bands of `telltale_query_duration_seconds` (50 µs to 5 s, and slower), darker where more answers landed; one path or several (**cache**, **upstream**, **local names**, **blocked & special**). The dashed line is the [latency objective](#service-level-objectives)'s threshold. Dots are real queries (exemplars): select one to open it in the query log. The ticks along the top are [change markers](#change-markers). **Table** lists the counts. - On the **Upstreams** page, **Answer time over time** shows the same for one upstream. - `GET /api/v1/stats/latency/heatmap?from=-6h&by=path|upstream&step=minute|hour` (minutes back 48 hours, hours back 400 days) and `GET /api/v1/stats/latency/exemplars` (the last 512 per node, newest first; agents need `querylog:read`) return the data; the MCP tool `latency_breakdown` with `histogram: true` adds the last hour's bands. Each node counts its answers into the bands as it already does for `/metrics`, once per minute bucket; older builds' minutes have no bands and the answer says `partial`. ### Heatmap **Monitor → Heatmap** shows when queries happen: one row per day (or, with **Typical week**, one per weekday, each cell the mean of those days), one column per hour, 15 minutes, or minute of the day, in your browser's local time. - **Network** counts every query exactly, from the dashboard's per-minute and per-hour counts: hours back 400 days, 15-minute and minute slots back 7 days. **Blocked** colours blocked queries instead. - **Device** (an address) and **Name** come from each hour's top lists (the busiest 100), by hour. While it's in the list the count is exact; hatched hours had it below the list, so its count there is unknown, not zero. Open one from the Clients page (**When**) or the Traffic map. - Point at a cell (or move with the arrow keys) for its count; select it (or press Enter) to open the query log from that time. The bars on the right and at the bottom sum each row and column; the small ticks are [change markers](#change-markers). **Table** shows it by hour. - `GET /api/v1/stats/heatmap?from=-7d&grain=15m` (`kind=client|name`, `key`, `measure=blocked`) and the MCP tool `activity_heatmap` return the same as parallel arrays. A name needs privacy level 0, a device level 1 or lower. ### Flow **Monitor → Flow** shows where queries go, as one diagram: **groups** → **decision** (cached, forwarded, blocked, local, …) → **upstream** → **response code**. The width of each band is the number of answers, and red bands end in a SERVFAIL. - Every band is a count kept for exactly that pair as queries were answered, never an estimate: each minute counts group → decision for every answer, then decision → upstream → response code for the answers an upstream was asked for, and decision → response code for the rest. A dropped query has no response code, so it ends at its decision. - **Network** reads those minute counts, every node of a cluster (15 minutes to 7 days). Minutes from a build before 0.5 have no flow counts: the card says some answers are missing. - **Device** (an address) reads that device's queries in the query log, at most the last 24 hours. If the read stops early, the card says so. It needs query-log privacy level 1 or lower. - Point at a band for its count and share (of every answer, and of where it comes from); select a node to open what's behind it (a group's settings, the lists for **blocked**, the upstreams, or the query log for a decision or a response code). Nodes under 0.5 % fold into **other** until you tick **Show small**. **Live** refreshes it every 15 seconds and shows when it last did. **Table** lists every band. On a phone the stages run top to bottom. - `GET /api/v1/stats/flow?from=-6h` (`client` for one device) and the MCP tool `query_flow` return the nodes and links. Grafana has no built-in Sankey panel; a community Sankey plugin can draw the same from that endpoint. ### Wall display `#/wall` (the **Wall display** button at the top right, between pause and the theme, in the Advanced view) is a full-screen page for a monitor on the wall or a tablet on a shelf: no menu, one panel at a time on a timer, and the live query stream as a ticker along the bottom. - Panels: `map` (this hour's busiest names as tiles that move as they grow), `flow` (the last hour's query flow, with dots travelling along the bands), `slo` (each service-level objective's budget left as a ring, the last hour's burn rate under it (amber from 1×, red from 6×), and the devices that spent it), `latency` (the last hour's answer-time heatmap), and `heat` (the network's week). - The address chooses them: `#/wall?panels=slo,flow&every=30` shows those two, in that order, 30 seconds each (default: all six, 20 seconds; 5 to 600). The sixth, `kpi`, is the "right now" tile: the last hour's queries per second, blocked share, answers from the cache, and the health word. - **The layout fits the screen** (`?layout=auto`, the default), from the wall's size: - a phone or small tablet (shorter side under 600 px): one panel at a time, as before; - a portrait screen: two panels stacked; - a TV (16:9 or 16:10): a large panel and two small ones beside it (the "right now" tile and the service level); - an ultrawide screen (more than 2.1 × as wide as tall): a large panel in the middle and four around it; - a large screen (2560 × 1300 and up): all six at once, nothing rotating, and the live queries as a column beside them. Panels that don't fit rotate in the large panel every `every` seconds; the arrow keys move it. `?layout=single` keeps one panel at a time on any screen, `?layout=grid` the large screen's grid, and `?layout=portrait` (or `compact`, `standard`, `ultrawide`, `large`) forces one. A wall display's link can carry it. `?motion=off` (or the system's reduced-motion setting) stops every animation; the panels still change. - **Space** pauses, the arrow keys move between panels, **Esc** or a click goes back to the dashboard. Nothing animates while the tab is hidden. - It needs a signed-in session like any page. For a screen nobody sits at, make it a **wall display** instead (**Settings → Users → Wall displays**, admins): give it a name and, best, the screen's address or network (`192.168.2.40`, `192.168.2.0/24`), and open the link it shows once on that screen's browser. The screen then stays signed in until you revoke it (or until the days you set run out), with no idle sign-out, but it can only show the wall: the traffic map, the flow, the service level, the heatmaps, the change markers, and the live queries (it does see live query names, like a viewer). Every other page and every change is refused, and from an address outside its networks it's refused too. The link is shown once; making, opening, and revoking one is in the audit log. In a cluster, make it on the primary; every node accepts it. `POST /api/v1/displays` makes one from a script; the code only works as a link, never as an API token. - The Grafana dashboard is the other wall. ### Change markers Every time chart (the dashboard's, the Cache page's) draws a thin line for each thing that was done to a node, or that the node did to itself, so a change lines up with what it did to the graph: | Marker | When it's written | |---|---| | Configuration applied | a change took effect: its version, where it came from (`ui` for a signed-in user, `api` for an API token, `agent`, `gitops`, `file` when the config files were reloaded), who made it, and which sections changed. On a cluster replica: the version, and `gitops` or `cluster` (a replica can't tell who; the primary's marker says) | | Build started, node restarted | every start, and a new build the first time it starts (`uncleanShutdown` after a crash, an out-of-memory kill, or a power cut) | | Lists refreshed | a list download changed the rules: which lists, and how many blocked names were gained or lost (a refresh that changed nothing writes none) | | Rollout | a [staged rollout](#rolling-a-change-out-in-stages) reached its canaries, was promoted, failed and was pinned back, aborted, pinned, or unpinned (the primary's) | | Maintenance, pauses, cache flushes | [maintenance](#taking-a-node-down-for-maintenance) entered or left; blocking paused or resumed; the cache flushed (the node asked, naming the nodes it applied to) | | Upstream benched or recovered | an upstream was benched (its circuit breaker opened) or is answering again (at most one per upstream every 5 minutes) | | Note | what an operator or an agent wrote (below) | - **Hover** a marker (or tab to it) for what happened, when, who, and on which nodes; select it to open where it happened (Settings, Cluster, Lists, Upstreams). One change applied on several nodes is one marker naming them. Where more than 40 would crowd a chart, close ones share a marker with a count (`×3`). - **Changes** in the dashboard header turns them off and on (remembered in this browser). In the Advanced view, chips pick the kinds. The node picker shows one node's markers only. - **Settings → System → Changes** lists the last 200 from every node, filterable: the table view of every chart's markers. `GET /api/v1/annotations?from=-24h&kinds=config.applied,note` returns them (newest first, at most 2,000 a page, `nextBefore` for the next page), and the MCP tool `list_annotations`; `get_overview` includes how many fell in its window and the latest three. - **Notes:** the **Note…** button (operators) puts your own marker on every chart of every node: a new access point, an ISP outage, a migration. 1 to 200 characters on one line, now or up to 7 days back. `POST /api/v1/annotations` with `{"text": "moved the AP", "time": "2026-10-16T14:05:00Z"}` does the same, and so does the MCP tool `annotate` (agents need the `ops:annotate` scope, which isn't in the default set). Notes are audit-logged (`annotation.note`) and can't be edited or deleted: add a correcting note. They aren't configuration, so a GitOps-managed cluster takes them too. - Each node keeps its own in `/annotations.jsonl`: at most 20,000 lines and 90 days, trimmed at start and once a day. They're written beside the work they describe, never on the query path. If the file can't be written (a full or read-only disk), the node logs it once an hour, counts it in `telltale_annotation_write_failures_total`, and carries on. - **Prometheus and Grafana:** `telltale_config_seq` is the configuration sequence the node applied (the cluster's, or a standalone node's configuration version), and `telltale_annotations_total{kind}` counts the markers written. The Grafana dashboard draws `changes(telltale_config_seq[5m]) > 0` (configuration), `changes(telltale_filter_snapshot_version[5m]) > 0` (lists), and `changes(telltale_process_starts_total[5m]) > 0` (restarts) as annotations on every panel, with no Grafana credentials. ### Host resources Every node samples its machine every 15 seconds (off the DNS path, from `/proc`, `/sys`, and the cgroup files) and shows it on the **Cluster** page, one card per node, with the last hour as a line under each bar. A standalone node shows its own card there too. In a cluster, the samples travel in the nodes' heartbeats, so any node's page shows every machine. | What | Where it comes from | Metric | |---|---|---| | Memory: total, available, swap | `/proc/meminfo` | `telltale_host_memory_bytes{kind}` | | The container's memory limit and use, OOM kills | cgroup v2 (`memory.max`, `memory.current`, `memory.events`) or v1 | `telltale_cgroup_memory_bytes{kind}`, `telltale_cgroup_oom_kills_total` | | CPU busy, load average, cores | `/proc/stat`, `/proc/loadavg` | `telltale_host_cpu_busy_ratio`, `telltale_host_load{window}`, `telltale_host_cpus` | | The container's CPU quota, and how often it throttles | `cpu.max`, `cpu.stat` (v1: `cpu.cfs_*`) | `telltale_cgroup_cpu_quota_cores`, `telltale_cgroup_throttled_ratio` | | Free space where the data lives; query log and list sizes; write rate | `statvfs` of `data_dir`; `/proc/self/io` | `telltale_data_filesystem_bytes{kind}`, `telltale_data_bytes{kind}`, `telltale_write_bytes_per_second` | | Temperature (the hottest sensor; a Pi's SoC) | `/sys/class/thermal` | `telltale_host_temperature_celsius` | | Open files against the limit, threads | `/proc/self` | `telltale_open_fds`, `telltale_max_fds`, `telltale_threads` | | OS, kernel, architecture, machine uptime | `/etc/os-release`, `/proc` | `telltale_host_info{os,kernel,arch}`, `telltale_host_uptime_seconds` | | Clock offset between nodes | heartbeat timestamps and round-trip time | (Cluster page) | The card says what looks wrong, and the Cluster page's `host_resources` check turns red with it: - the data disk 90% full or more; - the host's memory, or the container's limit, 90% used or more; - an out-of-memory kill in the last hour; - 80 °C or more; - a clock 2 s or more away from this node's. With `prometheusRule.enabled`, the chart adds matching alerts: - `TelltaleDNSDataDiskLow` (under 10% free for 10 minutes); - `TelltaleDNSMemoryPressure` (over 90% of the limit); - `TelltaleDNSOutOfMemoryKill`; - `TelltaleDNSCpuThrottled` (over 25% of periods); - `TelltaleDNSRunningHot`. `deploy/helm/alerts-test.sh` checks them with `promtool`. **What a node can't read is left out, not shown as zero:** - the `FROM scratch` image has no `/etc/os-release`; - most PCs and VMs have no temperature sensor; - outside a container there's no cgroup limit. The card lists what's unavailable. Older nodes in a mixed-version cluster simply show no card. **Grafana:** import `deploy/grafana/telltale-dashboard.json` (service level, traffic by status, answer-time percentiles, where time goes, upstream latency/share/failures/breakers, blocks by list and group, cache, top clients, and the health of TelltaleDNS itself). Pick the Prometheus data source, job, and instance at the top. **History:** once a minute, completed minutes are saved to `/rollups.db` (SQLite): per-minute counts for 7 days, per-hour for 400 days, per-day forever, and each hour's top domains, blocked names, NXDOMAIN names, clients, and latency percentiles. The dashboard's 7- and 30-day views, `GET /api/v1/stats/timeseries?step=hour|day`, and long `stats/summary` ranges read them, and they survive restarts. If the file can't be opened, history is limited to the 48 hours kept in memory; DNS is unaffected. ### Service-level objectives Two promises about DNS, each with an **error budget**: the bad answers it still allows over a window (30 days by default). ```toml [slo] enabled = true availability_target = 99.9 # percent of answers that aren't SERVFAIL latency_target = 99 # percent of answers sent within latency_ms latency_ms = 250 # 1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, or 5000 window_days = 30 # 1 to 90 ``` - **What counts:** availability counts every answer (any response code) and the SERVFAIL ones; latency counts every query that got an answer and the ones slower than `latency_ms`. Queries dropped without an answer (rate limits, junk) count for neither. DNSSEC failures and domains broken at their own servers are SERVFAIL too: that's what your devices saw. - **Burn rate:** how fast a budget is being spent. 1 spends exactly the budget over the window; 14.4 spends 2% of a 30-day budget in an hour. Each objective shows it over 5 minutes, 30 minutes, 1 hour, 6 hours, and 3 days. - **When it warns:** a long and a short window both burn too fast (the Google SRE workbook's pairs): **fast**, 14.4× over 1 hour and 5 minutes; **slow**, 6× over 6 hours and 30 minutes; **ticket**, 1× over 3 days and 6 hours (shown only). Fast and slow turn the [health](#health) icon amber (`slo_burn`) and fire `slo_burn` [alert rules](#alerts). A pair needs at least 50 answers in its short window and 10 bad answers in its long one, so a single SERVFAIL on a quiet network doesn't warn. - **Where:** the dashboard's **Service level** card (in the Advanced view: the Simple | Advanced switch at the top; in the menu on a phone), `GET /api/v1/stats/slo`, and the MCP tool `slo_status`. In a cluster the objectives cover every node's answers. They're worked out from the per-minute and per-hour counts the dashboard already keeps, so they cost nothing on the query path; slow answers are counted in the rollups from this release on (older hours count none). - **Prometheus:** `telltale_slo_objective_ratio{slo}` and `telltale_slo_latency_threshold_seconds` publish the settings. With `prometheusRule.enabled`, the Helm chart adds recording rules (`telltale:slo_availability_errors:ratio_rate5m` … `_3d`, and `telltale:slo_latency_errors:…`) and alerts (`TelltaleDNSSLOAvailabilityBurnFast`, `…BurnSlow`, `…BurnTicket`, and the same for latency) with the same windows and minimums; set `prometheusRule.slo` to the same targets as `[slo]`. Without Kubernetes, load `deploy/prometheus/telltale-slo-rules.yaml` (the same rules, for a scrape job named `telltale-metrics`). The Grafana dashboard's **Service level** row shows the SLIs, the budget left over the dashboard's time range, and the burn rate. #### Who spent it? **Who spent it?** under each objective on the Service level card (or **Analyze → Error budget**) breaks the window's bad answers down: - **Budget left** over the window as a line (one per node below it), with the [change markers](#change-markers), so a drop lines up with what changed. - **Clients** and **names** that got the bad answers: approximate top lists (each count may be up to its "±" over, Space-Saving's bound), kept per hour for 400 days. Select one to open its bad answers in the query log, where the log still has them (it keeps fewer days than the window). - **Upstreams** (`none`: no upstream on the answer, such as a SERVFAIL served from the cache), **status** (`refreshed`, `forwarded`, ...), **response code**, and **nodes**: exact, from the per-minute counts. - In a cluster every node is asked. A node on an older build, or a resolver pod that's gone (counted from the minutes it shipped, 7 days), adds to the totals without client and name lists, and is named as partial. The lists follow the query log's privacy level: names as hashes at 1, no clients at 2, only the timeline at 3. - `GET /api/v1/stats/slo/attribution?objective=availability&from=-30d` returns it, and the MCP tool `slo_attribution`. Counting it costs nothing on the query path: the analytics thread counts bad answers only. ### Listener checks A listener can be bound and still not answer: a stuck TLS stack, a firewall rule, a load balancer in front of it. So every node asks each of its own DNS listeners a question the way a device would, through the listener's own protocol: ```toml [probe] enabled = true interval_secs = 30 timeout_ms = 2000 cert_warn_days = 14 # a certificate expiring sooner degrades the health level targets = ["udp://192.168.5.112:53"] # extra: by IP, e.g. a load balancer's address ``` - **What's asked:** `probe.telltale.invalid`, type A, which TelltaleDNS answers NXDOMAIN itself (no upstream). Any well-formed answer counts. A listener bound to `0.0.0.0` or `[::]` is asked on loopback; DoH on its `path`. Targets take `udp://`, `tcp://`, `tls://`, `https://…/dns-query`, `h3://`, and `quic://` with an IP address. Certificates aren't checked by name (probes connect by IP); a listener with `proxy_protocol = true` is skipped, since probes don't send the PROXY header. - **Certificates:** each DoT/DoH/DoH3/DoQ listener's certificate file is read for its expiry, its SHA-256 fingerprint, and the names it covers. - **The web UI:** with `[api] tls`, the web UI and API are checked too (`proto="api"`): a `GET /livez` over HTTPS, and their certificate like a listener's. A self-signed one shows as such (it doesn't expire); compare its fingerprint on this page with the one your browser shows before trusting it. Your own certificate's expiry warns and alerts like a DNS listener's. - **Health:** a probe that fails twice in a row degrades the level (`probe_failing`); when every DNS listener of a node fails, the node isn't serving DNS (`not_serving`, severe unless other nodes serve; the web UI's check doesn't count toward it). A certificate within `cert_warn_days` degrades (`cert_expiring`); an expired one is severe (`cert_expired`), since encrypted clients can't connect. - **Alerts:** `probe_failing` and `cert_expiring` rules (below); with `prometheusRule.enabled`, the Helm chart adds `TelltaleDNSListenerNotAnswering` (2 minutes) and `TelltaleDNSListenerCertExpiring` (under 14 days). - **Where:** **Settings → System → Listener checks**, `GET /api/v1/system/probes` (every node), the MCP tool `probe_status`, and `/metrics`: `telltale_probe_success{target,proto}`, `telltale_probe_duration_seconds`, `telltale_probe_failures_total`, and `telltale_listener_cert_expiry_timestamp_seconds{target}`. - **Cost:** a handful of queries a minute, from a background task. They count in `telltale_queries_total` like any query, but stay out of the query log, the dashboard's top lists, device analytics, and event sinks. dnstap still sees them as client queries from loopback. ### Upstream truth checks Are your upstreams telling the truth? The **Upstreams** page's **Answer quality** card (and `GET /api/v1/analytics/upstream-checks`, the MCP tool `upstream_checks`) shows, per node and upstream since start: - **DNSSEC verdicts** on its answers, when `[dnssec] mode` validates: secure, unsigned, bogus, undecided. An upstream with bogus answers others don't have may be tampering, or broken. - **Error codes** it attached to its answers (Extended DNS Errors, RFC 8914). EDE 15 (Blocked), 16 (Censored), and 17 (Filtered) mean the upstream filters names itself: expected from a family filter, a surprise from a "plain" resolver. - **Second opinions** (off by default): ```toml [upstream_check] sample_every = 1000 # one forwarded question in 1,000 is asked again (0 = off) # reference = "quad9-only" # an upstream group to ask; default: another member of the same group keep = 100 # disagreements kept ``` After the device has its answer, a background task asks another upstream the same question and compares: **same**; **other addresses** (both had addresses, none in common: CDNs and geo-DNS answer differently per resolver, usually harmless); **other response** (a different response code); **filtered** (one side had addresses, the other none, or only `0.0.0.0`, `::`, or loopback: one of them filters that name); **no second answer**. The latest disagreements are listed with both answers; names follow the query log's privacy level. A group with a single upstream gets no second opinions unless `reference` names another group. At most 8 run at once; more are skipped. - **Privacy:** the second upstream sees the sampled names, which is why it's off by default. One in 1,000 adds 0.1% to upstream traffic. - `/metrics`: `telltale_upstream_dnssec_total{upstream,result}`, `telltale_upstream_ede_total{upstream,code}`, `telltale_upstream_check_total{upstream,result}`. - Cost: on the path to an upstream only (never for cached or blocked answers): one look for an EDE in each upstream answer and a counter per DNSSEC verdict. ### Query events Besides counters, every query also produces a detailed **event**: the time, the client and its group, the name and type, the outcome, the response code, which list and rule blocked or allowed it, and the timings. Every upstream exchange produces one too. Events feed the query log, top lists, and per-client analytics, which the web UI and the API (`GET /api/v1/queries`, `/api/v1/stats/...`) read. - Each thread writes events into its own buffer without waiting or locking, and a background thread collects them every 25 ms. If a buffer ever fills, events are dropped and counted, and DNS answers are never held back. Counters and the metrics above never drop. - `[telemetry] ring_slots` sets the buffer size per thread, in events of about 128 bytes (default `4096`, 512 KiB). That's about 170 ms of a fully loaded worker. Raise it if `telltale_telemetry_dropped_total` ever grows. - Collected so far, in memory: per-second counts for the last 15 minutes and per-minute counts for the last 48 hours; for the current and previous hour, the top domains, blocked domains, NXDOMAIN names, clients, and each recent client's top domains; and latency percentiles by path, query type, client, and upstream. | Metric | What it tells you | |---|---| | `telltale_telemetry_events_total{ring}` | events written, per thread name (`udp--` for the UDP workers; Tokio's workers share `telltale-rt` and are summed) | | `telltale_telemetry_dropped_total{ring}` | events dropped because a buffer was full (should stay 0) | | `telltale_clock_steps_total`, `telltale_clock_last_step_seconds` | times the event clock followed a step of the system clock (a Pi setting its clock from NTP after TelltaleDNS started, a resume from suspend), and the latest step. Events, the query log, and the rollups stay on the wall clock across a step. | ### Query log Every query event is also written to the **query log** on disk, in `/qlog/YYYY/MM/DD/HH--.seg`: one file per hour, compressed by column, about 14 bytes per query (50 million queries take about 690 MB). ```toml [telemetry.qlog] enabled = true retention_days = 30 # delete older hours retention_bytes = "2GiB" # and the oldest hours beyond this total privacy_level = 0 # 0 full; 1 hide domains; 2 hide domains and clients; 3 no per-query log flush_interval_secs = 10 # write at least this often (a crash loses at most this much detail) fsync = false # true: sync every write (slower on SD cards) ``` - Writes are sequential and batched, about one every 10 seconds on a quiet network, so it's gentle on SD cards. A slow or full disk never slows DNS: if the writer falls behind, rows are dropped and counted (`telltale_qlog_rows_dropped_total`). The store reports on itself too: `telltale_qlog_block_write_seconds` (the time block writes take; `rate(..._sum) / rate(..._count)` is the average), `telltale_qlog_segments` (segment files on disk), and `telltale_qlog_write_errors_total`. - `privacy_level = 1` stores each name as a hash, so identical names still group together but can't be read back; `2` also drops client addresses; `3` keeps no per-query log at all (counters, top lists, and graphs still work). - What the levels cover: everything TelltaleDNS keeps or shows, as on Pi-hole. The query log, the live tail, [event sinks](#event-sinks), the dashboard's top lists, a device's page, and the hourly top lists kept in `rollups.db` all show hashed names at 1 and above, and no client addresses at 2 and above (every query then counts as one client). Anomaly detection and new-device alerts are off at 1 and above: their findings name domains and devices. The masked-client warning is off at 2 and above. Counts, graphs, and latency keep working at every level. - What level 1 protects against: the hash is keyed with a random key kept in `/privacy.key` (in a cluster, the primary's, which every node adopts so logs group the same everywhere). Someone holding the log but not the key can't check whether a name they suspect was queried. Anyone with the key and the log can; keep the data directory private, or use level 3. - Search it in the web UI's **Query log**, through the API (`GET /api/v1/queries`), or from the command line: ```sh telltale qlog search doubleclick -c telltale.toml # names containing "doubleclick" telltale qlog search --client 192.168.1.50 --status blocked --since 3600 telltale qlog search '*.example.org' --match glob -t AAAA --json # JSON lines telltale qlog search --min-ms 500 --limit 20 # slow queries ``` Results come newest first; the last line prints a `--cursor` for the next page. `--match` is `substring` (default), `exact`, `suffix` (the name or anything below it), `glob`, or `regex`. - Searching is fast because it looks at the list of names first and skips whole files that can't match: on a Raspberry Pi 4, finding a rare name in 50 million queries over 30 days takes about 0.2 s, and the slowest searches about 2 s. Searches use up to 4 threads at the lowest CPU priority, so they never slow DNS down. - Metrics: `telltale_qlog_rows_written_total`, `_rows_dropped_total`, `_bytes_written_total`, `_segments_removed_total`, `_write_errors_total`. ### Keeping names and devices out of the log Noise drowns out what matters: phones checking their connection, time servers, a monitoring probe asking every few seconds. Keep them out of the log and the statistics: ```toml [exclusions] enabled = true # off keeps the lists, logs everything again names = ["connectivitycheck.gstatic.com", "ntp.org"] # each with its subdomains clients = ["192.168.1.10", "10.20.0.0/24"] # addresses or networks ``` - Their queries are still answered as usual and still counted in Prometheus (`telltale_queries_total`), but left out of the query log, the live view, the dashboard and top lists, device pages, anomalies, the event sinks, OTLP traces, and exemplars. `telltale_queries_excluded_total` counts them. - The dashboard's totals leave them out, so they'll be lower than Prometheus's. - From the next query on, on every node of a cluster (`[exclusions]` is shared). Queries logged before stay (and the top lists rebuilt after a restart skip them too). - In the UI: **Settings → Query handling → Kept out of the log**, with a toggle, checked before it's applied, and *Revert to the file*. Under a Git-managed configuration, change `[exclusions]` in Git; the UI shows the TOML to commit. API: `PUT /api/v1/exclusions/default`; MCP: `plan_set_exclusions`. ### Internal names Some names are your own network talking to itself: reverse lookups of private addresses (`20.1.168.192.in-addr.arpa`) and, in Kubernetes, the cluster's names (`postgres.db.svc.cluster.local`). A pod's search list alone turns one lookup of `api.github.com` into three failed ones under `cluster.local` first. These **internal names** get their own top list, so the dashboard's **Top domains** shows what your network reaches outside: - The dashboard's **Top domains** card switches between **Internet** and **Internal**. The NXDOMAIN list leaves internal names out too. - They're still in the query log, the live view, a device's own top domains, the blocked list, exports, and Prometheus: nothing is hidden, only sorted. - Internal by default: reverse lookups of private and local addresses (the same ranges as `[special] private_ptr_nxdomain`), and `cluster.local` with `[kubernetes] enabled`. Add your own, each with its subdomains, per node (`[telemetry]` isn't shared): ```toml [telemetry] internal_names = ["lan", "home.arpa"] # a Kubernetes cluster domain other than cluster.local goes here too ``` - API: `GET /api/v1/stats/top?kind=internal`; MCP: `top_items` with `kind: internal`. At privacy level 1 and above the list holds hashed names, sorted the same way. Top lists rebuilt from a privacy-hashed query log after a restart count those names as domains. ### OpenTelemetry (OTLP) Push the metrics to an OpenTelemetry collector (the OpenTelemetry Collector, Grafana Alloy, a hosted backend) instead of, or as well as, having Prometheus scrape them: ```toml [telemetry.otlp] endpoint = "http://otel-collector:4318" # OTLP/HTTP; metrics go to /v1/metrics interval_secs = 60 # headers = { "x-api-key" = "..." } # for hosted collectors ``` - The metric set is exactly what `/metrics` serves: counters become cumulative monotonic sums, gauges stay gauges, and histograms keep their buckets. Resource attributes: `service.name = telltaledns`, `service.version`, `host.name` (the node name). - Encoding is OTLP/HTTP with JSON, which every OTLP receiver accepts. - For **query events as OpenTelemetry logs**, add an event sink with `format = "otlp_logs"` (below), pointed at `/v1/logs`. Each record has a one-line body (`A example.com from 192.168.1.5: blocked`) and attributes (`dns.question.name`, `dns.question.type`, `dns.response_code`, `client.address`, `client.name`, `telltale.status`, `telltale.list`, ...); blocks are `WARN`, the rest `INFO`. - A collector that's down costs only that interval's points; DNS never waits for it. ### Exemplars and traces From a latency graph to the query behind it. - **Exemplars:** each bucket of `telltale_query_duration_seconds` carries an *exemplar*, a recent query that landed in it (its trace ID, its latency, and when it happened). They're in the OpenMetrics format, which Prometheus asks for by default. Prometheus keeps them when started with `--enable-feature=exemplar-storage`, and Grafana then shows them as dots on the latency panels (the bundled dashboard turns them on). Each bucket's exemplar is replaced at most once a second, so it's a recent example from that bucket, not the slowest query. `[telemetry.metrics] exemplars = false` turns OpenMetrics off; `/metrics` then always answers the plain Prometheus text format. - **From the dot to the query:** in Grafana's Prometheus data source, under *Exemplars*, add a link with label name `trace_id` and URL `https:///#/queries?trace=${__value.raw}` (or point it at your Tempo data source when traces go there). - **The query behind a trace ID:** open the query log with `#/queries?trace=` (or `GET /api/v1/queries?trace=`, the MCP tool `search_queries` with `trace`). The ID holds the query's start time and a hash of its query-log fields, keyed with the cluster's privacy key, so it finds the row on whichever node logged it and can't be used to confirm a guessed name. - **Traces:** send some queries to your OpenTelemetry collector as traces (Grafana Tempo, Jaeger, and the like): ```toml [telemetry.otlp] endpoint = "http://otel-collector:4318" # traces go to /v1/traces traces_sample_every = 1000 # 1 query in 1,000 (0: none, the default) traces_slow_ms = 250 # and every query that took at least 250 ms (0: none) ``` Each query is a server span (`DNS A`, with `dns.question.name`, `dns.question.type`, `dns.response_code`, `client.address`, `network.transport`, `telltale.status`, `telltale.group`, `telltale.node`) and, when upstreams were asked, a client span for the wait (`upstream quad9`, with `telltale.upstream` and `telltale.attempts`; it's placed at the end of the query, since the event records how long the wait was, not when it began). A SERVFAIL marks the span as an error. Its trace ID is the same as the exemplar's and the query log's. - Spans are built from the query event on the analytics thread and sent every 5 seconds, at most 512 queries per request. At most 4,096 wait; more are dropped, never slowing DNS. `telltale_traces_total{result="sent"|"dropped"|"failed"}` counts them. - Names and clients follow the query log's privacy level: names hashed at 1 and above, no client address at 2 and above. ### dnstap Stream client queries and responses, as the DNS messages themselves, to a dnstap reader (`dnstap-read`, `fstrm_capture`, Vector, Logstash, a network-security monitor): ```toml [telemetry.dnstap] socket = "/run/dnstap.sock" # a reader listening on a Unix socket, or: # address = "tcp://10.0.0.5:6000" sample_every = 1 # 1 = every query; 10 = one in ten buffer = 10000 # copies held while the reader is slow ``` - Each sampled query becomes a `CLIENT_QUERY` and a `CLIENT_RESPONSE` message with the client's address, the transport (UDP, TCP, DoT, DoH, DoQ), the times, and the full query and response messages. - Frame Streams with the usual handshake; if the reader isn't there, TelltaleDNS keeps retrying (every few seconds) and drops copies meanwhile. A reader that accepts the connection but stops reading is given up on after 5 seconds and reconnected, with a warning in the log. DNS never waits for it: copies go through a bounded queue (`buffer`), and with dnstap off the query path does nothing extra. Dropped copies are counted in `telltale_dnstap_dropped_total`. - **Upstream traffic too** (`forwarder = true` under `[telemetry.dnstap]`): each query TelltaleDNS sends to an upstream and its answer become `FORWARDER_QUERY` and `FORWARDER_RESPONSE` messages with the upstream's address, port, and protocol (UDP, TCP, DoT, DoH, DoQ, DNSCrypt), sampled like client messages. The built-in recursive resolver's own queries aren't included. - Read at startup: restart after changing it. ### Event sinks Copy every query event to your own log pipeline (Loki, Elastic, Splunk, Graylog, a SIEM) as it happens: ```toml [[telemetry.sink]] name = "archive" type = "file" # JSON lines, rotated path = "/var/log/telltale/queries.jsonl" max_bytes = "100MiB" # then queries.jsonl.1, .2, ... keep = 3 [[telemetry.sink]] name = "siem" type = "syslog" # RFC 5424 address = "udp://192.168.1.20:514" # or tcp:// (octet-counted frames), or tls://host:6514 # tls_ca = "/etc/telltale/siem-ca.pem" # tls://: a private collector's CA (public roots always work) facility = 16 # local0 statuses = ["blocked"] # only blocks; empty means every event [[telemetry.sink]] name = "loki" type = "webhook" # batched HTTP POST url = "https://logs.example.net/ingest" format = "json_lines" # or json_array, or otlp_logs (OpenTelemetry) token_file = "/run/secrets/ingest-token" # sent as "Authorization: Bearer " # token_scheme = "Splunk" # for Splunk HEC batch = 500 # events per POST flush_secs = 5 # or whatever arrived in 5 s # spill_max_bytes = "64MiB" # keep refused batches on disk until the collector is back ``` - Each event is the same JSON object as a row of `GET /api/v1/queries` (`time`, `client`, `clientName`, `group`, `name`, `qtype`, `status`, `rcode`, `proto`, `list`, `rule`, `totalMs`, `upstreamMs`, `answers`, `node`), with the query log's `privacy_level` applied (names hashed at 1, clients removed at 2). - Syslog messages are `1