Skip to content

Debugging Hardened Pods & Node Access

Kubernetes pods running with restrictive security contexts (AppArmor, seccomp, non-root, read-only rootfs) cannot be debugged with a simple kubectl exec. This page documents the patterns used to get a debug shell without loosening the production security posture.

Attaching a Privileged Sidecar

kubectl debug copies the pod and attaches a privileged container that shares the process namespace:

kubectl debug <POD> \
  --image=ubuntu \
  --share-processes \
  --copy-to=debugsa \
  --profile=sysadmin \
  -- sleep infinity

kubectl exec -it debugsa -c debugger-* -- /bin/bash

The sysadmin profile grants the debug container elevated privileges while leaving the original containers unchanged. --share-processes makes all pod processes visible inside the debug container.

Or use the pexec kubectl plugin:

kubectl pexec <POD> -it -T -- bash

Temporary Privileged Pods on a Node

For node-level debugging (kernel issues, device access, network namespaces), spawn a pod with hostNetwork: true, hostPID: true, and privileged: true pinned to a specific node via nodeSelector — typical images:

  • debian:stable for general shell access and raw network/USB inspection
  • nicolaka/netshoot for packet capture, traceroute, and DNS diagnostics

These pods are useful for inspecting /proc, raw network interfaces, or USB devices that the unprivileged production pods can't see.

Common Maintenance Operations

A small set of operator-side wrappers live outside the cluster and run periodically or on demand. Their effective behaviour:

Operation What it does
Helm upgrade audit Iterates every Helm release and compares chart version against the upstream repo
Longhorn label enforcement Reconciles recurring-job-group.longhorn.io/* labels on all PVCs (see Longhorn Backup Policy)
Zero-replica ReplicaSet cleanup Deletes ReplicaSets with spec.replicas: 0 across all namespaces
TV stack offline Scales down every Deployment in the tv namespace before PVC maintenance
Home Assistant + Frigate restart Rolling-restarts the two coupled deployments together
ModSecurity log tail Streams ingress-nginx error logs and filters for ModSecurity events

File Transfer to OpenWrt / dropbear Devices

Dropbear (used by OpenWrt) has no sftp-server. Use stdin/stdout for file transfers:

# Upload a file
ssh root@bpi-r4 'cat > /tmp/file' < localfile

# Never use scp — fails silently or with protocol error

Hardware Operation Failures

Hardware failures (EEPROM write, serial device access, SFP link, firmware flash, USB device probe) often have a specific root cause — blind retries can mask the real failure mode and waste time. The diagnostic approach:

  1. Enumerate possible root causes (counterfeit chip, wrong driver layer, kernel vs userspace mismatch, baud rate, voltage/jumper, wrong device node)
  2. Pick the most likely cause with a brief justification
  3. Confirm before attempting the next action