Debug a Kubernetes service locally

Install Telepresence with Homebrew/apt/dnf

Debugging a service locally with Telepresence

Imagine you have a service running in a staging cluster, and someone reports a bug against it. In order to figure out the problem you want to run the service locally... but the service depends on other services in the cluster, and perhaps on cloud resources like a database.

In this tutorial you'll see how Telepresence allows you to debug your service locally. We'll use the telepresence command line tool to swap out the version running in the staging cluster for a debug version under your control running on your local machine. Telepresence will then forward traffic from Kubernetes to the local process.

You should start a Deployment and publicly exposed Service like this:

$ kubectl run hello-world --image=datawire/hello-world --port=8000
$ kubectl expose deployment hello-world --type=LoadBalancer --name=hello-world

If your cluster is in the cloud you can find the address of the resulting Service like this:

$ kubectl get service hello-world
NAME          CLUSTER-IP     EXTERNAL-IP       PORT(S)          AGE
hello-world   10.3.242.226   104.197.103.123   8000:30022/TCP   5d

If you see <pending> under EXTERNAL-IP wait a few seconds and try again. In this case the Service is exposed at http://104.197.103.123:8000/.

On minikube you should instead do this to find the URL:

$ minikube service --url hello-world
http://192.168.99.100:12345/

Once you know the address you can store its value (don't forget to replace this with the real address!):

$ export HELLOWORLD=http://104.197.103.13:8000

And you send it a query and it will be served by the code running in your cluster:

$ curl $HELLOWORLD/
Hello, world!

Swapping your deployment with Telepresence

Important: Starting telepresence the first time may take a little while, since Kubernetes needs to download the server-side image.

At this point you want to switch to developing the service locally, replace the version running on your cluster with a custom version running on your laptop. To simplify the example we'll just use a simple HTTP server that will run locally on your laptop:

$ mkdir /tmp/telepresence-test
$ cd /tmp/telepresence-test
$ echo "hello from your laptop" > file.txt
$ python3 -m http.server 8001 &
[1] 2324
$ curl http://localhost:8001/file.txt
hello from your laptop
$ kill %1

We want to expose this local process so that it gets traffic from Kubernetes, replacing the existing hello-world deployment.

Important: you're about to expose a web server on your laptop to the Internet. This is pretty cool, but also pretty dangerous! Make sure there are no files in the current directory that you don't want shared with the whole world.

Here's how you should run telepresence (you should make sure you're still in the /tmp/telepresence-test directory you created above):

$ cd /tmp/telepresence-test
$ telepresence --swap-deployment hello-world --expose 8000 \
--run python3 -m http.server 8000 &

This does three things:

  • Starts a VPN-like process that sends queries to the appropriate DNS and IP ranges to the cluster.
  • --swap-deployment tells Telepresence to replace the existing hello-world pod with one running the Telepresence proxy. On exit, the old pod will be restored.
  • --run tells Telepresence to run the local web server and hook it up to the networking proxy.

As long as you leave the HTTP server running inside telepresence it will be accessible from inside the Kubernetes cluster. You've gone from this...

    graph RL
    subgraph  in Cloud
    server["datawire/hello-world server on port 8000"]
    end

...to this:

    graph RL
    subgraph Laptop
    code["python HTTP server on port 8000"]---client[Telepresence client]
    end
    subgraph  in Cloud
    client-.-proxy["Telepresence proxy, listening on port 8000"]
    end

We can now send queries via the public address of the Service we created, and they'll hit the web server running on your laptop instead of the original code that was running there before. Wait a few seconds for the Telepresence proxy to startup; you can check its status by doing:

$ kubectl get pod | grep hello-world
hello-world-2169952455-874dd   1/1       Running       0          1m
hello-world-3842688117-0bzzv   1/1       Terminating   0          4m

Once you see that the new pod is in Running state you can use the new proxy to connect to the web server on your laptop:

$ curl $HELLOWORLD/file.txt
hello from your laptop

Finally, let's kill Telepresence locally so you don't have to worry about other people accessing your local web server by bringing it to the background and hitting Ctrl-C:

$ fg
telepresence --swap-deployment hello-world --expose 8000 --run python3 -m http.server 8000
^C
Keyboard interrupt received, exiting.

Now if we wait a few seconds the old code will be swapped back in. Again, you can check status of swap back by running:

$ kubectl get pod | grep hello-world

When the new pod is back to Running state you can see that everything is back to normal:

$ curl $HELLOWORLD/file.txt
Hello, world!

What you've learned: Telepresence lets you replace an existing deployment with a proxy that reroutes traffic to a local process on your machine. This allows you to easily debug issues by running your code locally, while still giving your local process full access to your staging or testing cluster.


Now it's time to clean up the service:

$ kubectl delete deployment,service hello-world

Telepresence can do much more than this: see the reference section of the documentation, on the top-left, for details.

Install Telepresence with Homebrew/apt/dnf

Still have questions? Ask in our Gitter chatroom or file an issue on GitHub.

If this tutorial was useful to you, we'd appreciate it if you'd share it online.

results matching ""

    No results matching ""