Skip to content

Debugging Hardened Pods & Node Access

Kubernetes pods running with restrictive security contexts (AppArmor, seccomp, non-root, read-only rootfs) cannot be debugged with a simple kubectl exec. This page documents the patterns used to get a debug shell without loosening the production security posture.

Attaching a Privileged Sidecar

kubectl debug copies the pod and attaches a privileged container that shares the process namespace:

kubectl debug <POD> \
  --image=ubuntu \
  --share-processes \
  --copy-to=debugsa \
  --profile=sysadmin \
  -- sleep infinity

kubectl exec -it debugsa -c debugger-* -- /bin/bash

The sysadmin profile grants the debug container elevated privileges while leaving the original containers unchanged. --share-processes makes all pod processes visible inside the debug container.

Or use the pexec kubectl plugin:

kubectl pexec <POD> -it -T -- bash

Temporary Privileged Pods on a Node

For node-level debugging (kernel issues, device access, network namespaces), spawn a pod with hostNetwork: true, hostPID: true, and privileged: true pinned to a specific node via nodeSelector — typical images:

  • debian:stable for general shell access and raw network/USB inspection
  • nicolaka/netshoot for packet capture, traceroute, and DNS diagnostics

These pods are useful for inspecting /proc, raw network interfaces, or USB devices that the unprivileged production pods can't see.

Common Maintenance Operations

A small set of operator-side wrappers run periodically or on demand — some as in-cluster Windmill flows (e.g. Longhorn label enforcement), others as external scripts. Their effective behaviour:

Operation What it does
Helm upgrade audit Iterates every Helm release and compares chart version against the upstream repo
Longhorn label enforcement Reconciles recurring-job-group.longhorn.io/* labels on all PVCs (see Longhorn Backup Policy)
Zero-replica ReplicaSet cleanup Deletes ReplicaSets with spec.replicas: 0 across all namespaces
TV stack offline Scales down the tv-namespace Deployments that mount the CephFS/Ceph-RBD media PVCs (tv-v3-ceph, tv-finish-film-ceph, tv-finish-serie-ceph, tv-incomplete-ceph) before PVC maintenance
Home Assistant + Frigate restart Rolling-restarts the two coupled deployments together
ModSecurity log tail Streams ingress-nginx error logs and filters for ModSecurity events

Pre-Reboot Safety Gate

Node reboots and firmware flashes are where latent faults detonate — a config that drifted months ago is harmless until the next boot tries to apply it. Before any node reboot, a read-only gate script checks the invariants that make the reboot safe, and a non-zero exit means: fix the failures first.

  • Cluster state — etcd has 3/3 healthy members (so the reboot still leaves quorum), the target node is Ready with no stale drain/out-of-service taints, and all Longhorn disks on it are healthy.
  • Data safety — no replicated volume has its only running replica on the target node. Deliberately single-replica volumes are reported as expected downtime rather than failures, and an in-flight rebuild raises a warning.
  • Node-local invariants — over SSH, the mounts and labels required for a clean boot are verified: the Longhorn default-disk label and mount, the boot-time bind configs, CA bundle sanity, and free space on the OS state partition.

The gate mutates nothing, so it can run any time — including as a pre-flight sweep across all nodes before a rolling upgrade.

File Transfer to OpenWrt / dropbear Devices

Dropbear (used by OpenWrt) has no sftp-server. Use stdin/stdout for file transfers:

# Upload a file
ssh root@bpi-r4 'cat > /tmp/file' < localfile

# Never use scp — fails silently or with protocol error

Hardware Operation Failures

Hardware failures (EEPROM write, serial device access, SFP link, firmware flash, USB device probe) often have a specific root cause — blind retries can mask the real failure mode and waste time. The diagnostic approach:

  1. Enumerate possible root causes (counterfeit chip, wrong driver layer, kernel vs userspace mismatch, baud rate, voltage/jumper, wrong device node)
  2. Pick the most likely cause with a brief justification
  3. Confirm before attempting the next action