DNS Operations (Pi-hole + Unbound)

Architecture (HA)

                    ┌─ Pi-hole (pi-k3s:53) ──── Unbound (pi-k3s:5335) ──── Root Servers
User Device ───┤
                    └─ Pi-hole-secondary (pi5-worker-1:53) ──── Unbound-secondary (pi5-worker-1:5335) ──── Root Servers

Both paths are independent. Clients may use either Pi-hole instance via DHCP-assigned DNS.

Why Unbound? Full recursive resolution directly to authoritative DNS servers. Better privacy (no single upstream sees all queries), no third-party trust required, DNSSEC validation.

Service Details

Pi-hole (Primary + Secondary)

• Image: pihole/pihole:latest with hostNetwork: true
• Primary: Port 53 on pi-k3s (192.168.1.55)
• Secondary: Port 53 on pi5-worker-1 (192.168.1.56)
• Password: Synced from 1Password via ExternalSecret
• Upstream DNS: Respective Unbound ClusterIP service
• Adlists: Firebog curated lists via ConfigMap (~900k domains)
• DNSSEC: Disabled on Pi-hole (handled by Unbound)
• Caching: Pi-hole maintains its own DNS cache

Unbound (Primary + Secondary)

• Image: madnuttah/unbound:latest (distroless, minimal — no cat/ls/head)
• Port: 5335 (non-privileged) via ClusterIP service
• Type: Recursive resolver with DNSSEC validation
• Node Placement: Pi 5 nodes ONLY via nodeSelector (Pi 3 hardware causes TCP failures)
• Performance: ~21ms uncached, 0-15ms cached

Upstream transport — do NOT set tcp-upstream: yes. It is not a "fall back
to TCP if UDP fails" toggle (that fallback is automatic on truncated UDP) — it
forces all upstream recursion over TCP, always. We ran it Dec 2025 → Jun 2026
under that misconception (added in 17b857f as generic "home-network resilience";
the AT&T/IPv6 slowness was a separate gateway-side fix — UniFi IPv6 prefix
delegation, nothing to do with DNS transport). It caused recurring DNSMASQ_WARN
"max concurrent queries (150)" on both Pi-holes at the same instant, plus
CONNECTION_ERROR premature TCP drops to Unbound (the Jan 2026 TCP-pool bump
4ed431f only masked the symptom). Removed 2026-06-12. UDP-first + automatic TCP
fallback + the default edns-buffer-size: 1232 is the correct fragmentation-safe
setup. If UDP ever is proven broken, fix EDNS buffer sizing — don't force TCP.

Configuration

Pi-hole v6 API (Critical)

Pi-hole v6 ignores most environment variables. Configuration is done via REST API in postStart hook:

• POST /api/auth - Get session ID
• PATCH /api/config - Set upstream DNS to Unbound ClusterIP
• POST /api/lists - Add adlists from ConfigMap (batch format)
• POST /api/action/gravity - Update gravity database

Reference: docs/pihole-v6-api.md

madnuttah/unbound Config Quirks (CRITICAL)

The madnuttah/unbound image rewrites the main unbound.conf during its entrypoint. It transforms paths and drops directives it doesn't recognize.

• Compiled config path: /usr/local/unbound/unbound.conf (NOT /opt/unbound/etc/unbound/unbound.conf)
• Our mount: /opt/unbound/etc/unbound/unbound.conf → entrypoint processes this into the actual config
• Dropped directives: domain-insecure and other less common server directives are silently dropped
• conf.d directory: /usr/local/unbound/conf.d/*.conf — mount custom server directives HERE as separate files
• Verify running config: kubectl exec -n pihole deploy/unbound -- unbound-checkconf -o <directive>

To add custom server directives (e.g., domain-insecure):

1. Add the directive to the ConfigMap as a separate data key (e.g., dnssec-exceptions.conf)
2. Mount it at /usr/local/unbound/conf.d/<name>.conf using subPath
3. Do NOT put it in the main unbound.conf — it will be silently dropped

DNSSEC Validation

Unbound validates DNSSEC via:

• auto-trust-anchor-file — root trust anchor
• harden-dnssec-stripped: yes — strict mode, rejects responses that strip DNSSEC

When a domain has broken DNSSEC (DS records published but no valid DNSKEY):

• Unbound returns SERVFAIL
• Unbound logs show: validation failure <domain>: no keys have a DS with algorithm RSASHA256
• Fix: Add domain-insecure: "domain.com" via conf.d mount (see above)

Caching Behavior (CRITICAL for Diagnostics)

Unbound serve-expired configuration:

• serve-expired: yes with serve-expired-ttl: 86400 (24 hours)
• Unbound serves stale cache immediately while refreshing in background
• This means a broken upstream can be masked by stale cache for up to 24 hours

Pi-hole also caches results independently.

Combined effect: testdnsquery (which runs dig inside Pi-hole) can return a successful cached result even when Unbound is actively returning SERVFAIL. A successful testdnsquery does NOT prove the resolution path is healthy. Always use diagnose_dns for troubleshooting.

DNS Resilience

The Pi node uses static DNS (1.1.1.1, 8.8.8.8) configured via NetworkManager.

• Why: Ensures the Pi can pull images (like Pi-hole itself) even if the cluster DNS is down.

IPv6 Blocking (AT&T Routing Issues)

AT&T Fiber has poor IPv6 routing to some CDNs. We selectively block IPv6 for affected domains.

Current blocked domains: See clusters/pi-k3s/pihole/pihole-custom-dns.yaml

IMPORTANT: Test Before Blocking

When a user reports slow/broken connectivity to a service, DO NOT immediately add it to the IPv6 block list. First verify IPv6 is the cause:

# 1. Check if domain returns AAAA records (if no AAAA, IPv6 isn't the issue)
dig AAAA <domain>

# 2. Compare IPv4 vs IPv6 response times from a device on the network
curl -4 -w "IPv4: %{time_total}s\n" -o /dev/null -s https://<domain>
curl -6 -w "IPv6: %{time_total}s\n" -o /dev/null -s https://<domain>

# 3. If IPv6 is significantly slower (2x+) or times out, add to block list

Only add to the block list after confirming IPv6 is the problem. See docs/known-issues.md for details.

DNS Troubleshooting Runbook (MANDATORY)

When a user reports a domain is unreachable, follow ALL steps in order. Do NOT stop after step 1 even if it shows success. Do NOT blame the client's browser or machine until the entire server path is proven clean.

Step 1: Run diagnose_dns MCP Tool (ALWAYS START HERE)

Use diagnose_dns with the reported domain. This single tool tests:

• Pi-hole resolution (may be cached)
• Unbound primary direct resolution (bypasses cache)
• Unbound secondary direct resolution (bypasses cache)
• DNSSEC validation check (if Unbound fails, retries with +cd)
• Unbound pod logs filtered for the domain

If diagnose_dns is unavailable, manually run all substeps:

test_dns_query → check Unbound logs (BOTH pods) → dig @unbound directly

Step 2: Interpret Results

Step 3: Check Unbound Logs (NEVER SKIP)

Use getpodlogs for BOTH Unbound pods:

• namespace: pihole, pod: unbound
• namespace: pihole, pod: unbound-secondary

Look for:

• validation failure — DNSSEC issue
• SERVFAIL — upstream failure
• connection timed out — network issue
• TCP connection failed — TCP upstream issue (common on Pi 3)

Step 4: Apply Fix Based on Root Cause

Common Issues

1. "Refused": Check if Pi-hole is running and hostNetwork is true.
2. "ServFail": ALWAYS check Unbound logs. Often DNSSEC failure, upstream timeout, or TCP failure on Pi 3 nodes.
3. Adlists not loading: Check postStart hook logs in Pi-hole pod.
4. testdnsquery shows success but client fails: Stale cache. Use diagnose_dns instead.

Pi-hole Custom DNS — GitOps Managed (CRITICAL)

All local DNS records live in clusters/pi-k3s/pihole/pihole-custom-dns.yaml (ConfigMap). This ConfigMap is mounted by both pihole and pihole-secondary; Flux applies it to both instances simultaneously.

Current layout:

• Wildcard *.lab.mtgibbs.dev → 192.168.1.55 (ingress controller)
• Specific overrides below the wildcard (dnsmasq is most-specific-match-wins)

DO NOT add records via the Pi-hole web UI for anything cluster-managed — they will drift from GitOps and will not survive a pod restart. Always edit the ConfigMap and let Flux reconcile.

pi-k3s Master: Static DNS Fallback (CRITICAL — intentional design, DO NOT "fix")

The pi-k3s master node is configured with public DNS (1.1.1.1, 8.8.8.8) via NetworkManager. Worker nodes use Pi-hole.

Why: Bootstrap resilience. Allows pi-k3s to pull container images (including Pi-hole itself) even when the cluster DNS is down. This is intentional. Do not change it.

Side effect: pi-k3s does not resolve Pi-hole local overrides. Pods scheduled on pi-k3s that need a *.lab.mtgibbs.dev hostname get the public wildcard (192.168.1.55) instead of any Pi-hole-specific override. kubelet uses the host's resolv.conf, not CoreDNS.

Workaround for pi-k3s-local DNS names: Add /etc/hosts entries on the pi-k3s host for any local-only name that pods on that node need. These are managed under Flux at clusters/pi-k3s/coredns-custom/pi-k3s-hosts-overrides.yaml (a ConfigMap mounted by a DaemonSet or postStart hook — verify current implementation in the manifest).

Currently overridden: storage.lab.mtgibbs.dev → 192.168.1.61.

Full notes in memory/pi-k3s-dns-fallback.md.

CoreDNS lab.mtgibbs.dev Forwarding (CRITICAL — permanent infrastructure)

CoreDNS default config forwards ALL queries (including lab.mtgibbs.dev) to 1.1.1.1/8.8.8.8. The lab.mtgibbs.dev zone has a public wildcard record — so without a local override, all *.lab.mtgibbs.dev in-cluster DNS resolves to the public wildcard answer rather than the Pi-hole local records.

Fix: A lab.mtgibbs.dev.server block in the coredns-custom ConfigMap forwards local-domain queries to Pi-hole. This is under Flux at clusters/pi-k3s/coredns-custom/. Do NOT remove or edit it via kubectl.

If this block is missing, all *.lab.mtgibbs.dev in-cluster DNS breaks silently.

Full notes in memory/coredns-forward-gotcha.md.

Relevant Files

• clusters/pi-k3s/pihole/pihole-deployment.yaml — Primary Pi-hole
• clusters/pi-k3s/pihole/pihole-secondary-deployment.yaml — Secondary Pi-hole
• clusters/pi-k3s/pihole/unbound-deployment.yaml — Primary Unbound + Service
• clusters/pi-k3s/pihole/unbound-secondary-deployment.yaml — Secondary Unbound + Service
• clusters/pi-k3s/pihole/unbound-configmap.yaml — Unbound config (shared by both, includes conf.d entries)
• clusters/pi-k3s/pihole/pihole-custom-dns.yaml — Local DNS + IPv6 overrides
• clusters/pi-k3s/coredns-custom/ — CoreDNS lab.mtgibbs.dev forwarding + pi-k3s host overrides