Troubleshooting

Why can’t I SSH into my worker node?

What’s happening

You can’t access your worker node by using an SSH connection.

Why it’s happening

SSH by password is unavailable on the worker nodes.

How to fix it

To run actions on every worker node, use a Kubernetes DaemonSet, or use jobs for one-time actions.

To get host access to worker nodes for debugging and troubleshooting purposes, review the following options.

Debugging by using kubectl debug

Use the kubectl debug node command to deploy a pod with a privileged securityContext to a worker node that you want to troubleshoot.

First, Please setup the latest kubectl tool which supports debug (we are using kubectl v1.25 for debug)

To install kubectl, follow this web link: https://kubernetes.io/docs/tasks/tools/ and To Download Kubeconfig.yaml File and connect, kindly click here for more details.

The debug pod is deployed with an interactive shell so that you can access the worker node immediately after the pod is created. For more information about how the kubectl debug node command works, see debug command in Kubernetes reference.

  1. Get the name of the worker node that you want to access. The worker node name is its private IP address.

$ kubectl get nodes -o wide

  1. whichever node is required to be accessed, please run the following command

$ kubectl debug node/<NODE_NAME> --image=docker.io/library/alpine:latest -it

For Example

$ kubectl debug node/onekube-ip-10-5-12-6 --image=docker.io/library/alpine:latest -it

Here in example, onekube-ip-10-5-12-6 is the master node

  1. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as ip, ifconfig, nc, ping, and ps, are already available in the shell. You can also install other tools, such as mtr, tcpdump, and curl, by running apk add <tool>

Note

Almost all basic commands are there on the alpine container. You can also use some other container or even a full-fledged distro like ubuntu depending on your requirement.

Debugging by using kubect exec

If you are unable to use the kubectl debug node command, you can create an Alpine pod with a privileged securityContext and use the kubectl exec command to run debug commands from the pod’s interactive shell.

  1. Get the name of the worker node that you want to access. The worker node name is its private IP address.

$ kubectl get nodes -o wide

  1. Export the name in an environment variable.

$ export NODE=<NODE_NAME>

  1. Create a debug pod on the worker node. The Docker alpine image here is used as an example. If the worker node dosen’t have public network access, you can maintain a copy of the image for debugging in your own ICR repository or build a customized image with other tools to fit your needs.

$ kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: debug-${NODE}
  namespace: default
spec:
  containers:
   - args: ["-c", "apk add tcpdump mtr curl; sleep 1d"]
     command: ["/bin/sh"]
     image: icr.io/armada-master/alpine:latest
     imagePullPolicy: IfNotPresent
     name: debug
     resources: {}
     securityContext:

  1. Log in to the debug pod. The pod’s interactive shell is automatically opened. If the kubectl exec command fails, continue to option 3.

$ kubectl exec -it debug-${NODE} -- sh

You can use the kubectl cp command to get logs or other files from a worker node. The following example gets the /var/log/syslog file.

$ kubectl cp default/debug-${NODE}:/host/var/log/syslog ./syslog

Get the following logs to look for issues on the worker node.

/var/log/syslog
/var/log/containerd.log
/var/log/kubelet.log
/var/log/kern.log

  1. Run debug commands to help you gather information and troubleshoot issues. Commands that you might use to debug, such as ip, ifconfig, nc, ping, and ps, are already available in the shell. You can also install other tools, such as dig, tcpdump, mtr, and curl, by running apk add <tool>. For example, to add dig, run apk add bind-tools.

Note

Before you can use the tcpdump command, you must first move the binary to a new location that does not conflict with the install path for the binary on the host. You can use the following command to relocate the binary: mv /usr/sbin/tcpdump /usr/local/bin/.

  1. Delete the host access pod that you created for debugging.

$ kubectl delete pod debug-${NODE}