Skip to content

Debugging Hardened Pods & Node Access

Kubernetes pods running with restrictive security contexts (AppArmor, seccomp, non-root, read-only rootfs) cannot be debugged with a simple kubectl exec. This page documents the patterns used to get a debug shell without loosening the production security posture.

Attaching a Privileged Sidecar

kubectl debug copies the pod and attaches a privileged container that shares the process namespace:

kubectl debug <POD> \
  --image=ubuntu \
  --share-processes \
  --copy-to=debugsa \
  --profile=sysadmin \
  -- sleep infinity

kubectl exec -it debugsa -c debugger-* -- /bin/bash

The sysadmin profile grants the debug container elevated privileges while leaving the original containers unchanged. --share-processes makes all pod processes visible inside the debug container.

Or use the pexec kubectl plugin:

kubectl pexec <POD> -it -T -- bash

Temporary Privileged Pods on a Node

For node-level debugging (kernel issues, device access, network namespaces):

# Privileged Debian pod with host network on node 'qui'
./qui-tmp-debian-priv.sh

# nicolaka/netshoot with host network (network diagnostics)
./qui-tmp-netshoot.sh

These scripts spawn a pod with hostNetwork: true, hostPID: true, and privileged: true pinned to the qui node. Useful for inspecting /proc, raw network interfaces, or USB devices.

Common Maintenance Scripts

Script Purpose
./helm-available-upgrades.sh List available Helm chart upgrades across all releases
./longhorn-backup-labels.sh Apply/enforce Longhorn backup labels on all PVCs
./rs-clean-all.sh Delete zero-replica ReplicaSets across all namespaces
./tv-pvc-offline.sh Scale down TV namespace before PVC maintenance
./mdapi-restart-ha-frigate.sh Restart Home Assistant and Frigate deployments
./modsec.sh Tail ModSecurity WAF logs from ingress-nginx

File Transfer to OpenWrt / dropbear Devices

Dropbear (used by OpenWrt) has no sftp-server. Use stdin/stdout for file transfers:

# Upload a file
ssh root@bpi-r4 'cat > /tmp/file' < localfile

# Never use scp — fails silently or with protocol error

Hardware Operation Failures

When a hardware operation fails (EEPROM write, serial device access, SFP link, firmware flash, USB device probe), do not retry immediately:

  1. Enumerate possible root causes (counterfeit chip, wrong driver layer, kernel vs userspace mismatch, baud rate, voltage/jumper, wrong device node)
  2. Pick the most likely cause with a brief justification
  3. Confirm before attempting the next action

Blind retries on hardware waste time and can mask the real failure mode.