Debugging on Remote Kubernetes using Workstation Tools

What is my current challenge ?

95% of issues can be easily diagnosed with good telemetry data with logging, traces, and metrics. But there are some tricky issues that do require remote troubleshooting. The remote network, enterprise firewall rules make it a little tricky to leverage local tooling.

As an example, Have you tried running a remote debugger on a remote pod ? Or have you ever wanted to connect to a remote database/messaging system, such as Kafka, using a workstation but have been blocked by firewall rules?

What if there was an easier way to troubleshoot using tools that are setup on your local workstation ? What if you could use these tools to connect to remote systems and databases? Meet two of my favorite tools, sshuttle and kuttle which is essentially a poor man's VPN solution leveraging kubectl exec.

Pre-Requisites

• You already have kubectl configured to connect to remote cluster.
• You have access to launch a python image with shell using kubectl on remote cluster.
• You are leveraging Linux or MacOS or WSL2/VM on Windows.
• You have local administrator privilege's on local workstation. Admin privilege's on remote cluster is not needed.

How do I setup ?

• Setup sshuttle on local workstation
• Setup kuttle on local workstation
• Launch python shell container on K8s cluster. 

kubectl run kuttle -n kube-system --image=python:alpine --restart=Never -- sh -c 'exec tail -f /dev/null' (Note: You can change to desired namespace using -n option)

• Run sshuttle on your local workstation.

sshuttle -v -r 'kuttle -n kube-system' -e kuttle 10.121.0.0/16 -v

(Replace 10.121.0.0/16 by remote CIDR range for your k8s network)

• Voila! This should create a tunnel for you , and all traffic destined for 10.121.0.0/16 will be routed through this tunnel. Now you can use any workstation tooling, to connect to external database or external kafka or any pod within k8s cluster without having to change host file entries or ip addresses.

How does it work under the hood?

• When we launch sshuttle , it starts a process listening on port 12300 on your local workstation.
• sshuttle modifies the iptables on your machine, so that all traffic destined for 10.121.0.0/16 IP range, will now be routed to the sshuttle process listening on port 12300. When you look at the log entry, you can see sshuttle runs following iptables command, that essentially redirect all TCP traffic to local port 12300

>> iptables -t nat -A sshuttle-12300 -j REDIRECT --dest 10.161.0.0/16 -p tcp --to-ports 12300 -m ttl ! --ttl 42

• Sshuttle now launches kuttle under the hood, which will launch a python program called assembler on remote shell pod using kubectl exec. 

├─kubectl,5329 exec -i kuttle -n kube-system -- /bin/sh -c exec /bin/sh -c 'P=python3; $P -V 2>/dev/null || P=python; exec "$P" -c '"'"'import sys, os; verbosity=1; sys.stdin = os.fdopen(0, "rb"); exec(compile(sys.stdin.read(1317), "assembler.py", "exec"))'"'"''

• Sshuttle will take all TCP traffic on port 12300 and multiplexes to remote python program using kubectl exec session.
• Remote python program will assemble all the packets , and resend it to require destination. It will take response from the remote destinations, and send it back to sshuttle using the exec session.

Final word of thought

As powerful as this tooling might sound, it does allow remote Python script execution. For normal applications, this can easily be blocked by runtime container scanning tools. As a result, this approach should only be used in a break-glass scenario, when telemetry data was not enough to get to the root cause.