How to Debug Kubernetes Pods with Telepresence: Step-by-Step Tutorial

Debugging applications running in Kubernetes pods can be challenging and time-consuming. The traditional approach involves modifying code, rebuilding containers, redeploying to the cluster, and then checking logs—a process that significantly slows down development. Telepresence offers a powerful alternative by allowing you to run code locally while connected to your Kubernetes cluster, eliminating the need for constant redeployment cycles.

What is Telepresence and Why Use It?

Telepresence is a CNCF tool that creates a two-way network bridge between your local development environment and your Kubernetes cluster. This allows you to run a service locally while still connecting to other services in the cluster. The key benefits include:

Faster development cycles by eliminating container rebuilds and redeployments
Ability to use local debugging tools with services running in Kubernetes
Testing code against realistic production-like environments
Reproducing and fixing production issues more efficiently

Setting Up a Sample Application for Debugging

To demonstrate Telepresence's capabilities, we'll use a simple Go application with REST APIs and HTTP handlers. The application includes a deliberate error scenario that only occurs in production environments, making it an ideal candidate for demonstrating remote debugging capabilities.

// Data handler that simulates production errors
func dataHandler(w http.ResponseWriter, r *http.Request) error {
    // Generate random number to simulate intermittent errors
    rand.Seed(time.Now().UnixNano())
    randomNum := rand.Intn(10)
    
    // In production environment, throw error if number > 5
    if os.Getenv("ENV") == "production" && randomNum > 5 {
        return errors.New("error loading data")
    }
    
    // Return success response
    response := map[string]string{"message": "Data loaded successfully"}
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(response)
    return nil
}

The application includes a centralized error handling mechanism through a wrapper function called portLogger, which not only logs the hostname and port but also manages exceptions raised by the handlers.

The data handler simulates production errors by generating random numbers and throwing exceptions when specific conditions are met

Deploying the Application to Kubernetes

Before we can use Telepresence for debugging, we need to deploy our application to a Kubernetes cluster. Here's how to containerize and deploy the application:

Create a Docker image for the Go application
Prepare a Kubernetes deployment YAML with the ENV variable set to "production"
Create a Kubernetes service to expose the application
Apply the configuration to your cluster

YAML

# Deployment configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-app
  template:
    metadata:
      labels:
        app: go-app
    spec:
      containers:
      - name: go-app
        image: go-telepresence
        imagePullPolicy: Never
        env:
        - name: ENV
          value: "production"
        ports:
        - containerPort: 8080
---
# Service configuration
apiVersion: v1
kind: Service
metadata:
  name: go-app-service
spec:
  selector:
    app: go-app
  ports:
  - port: 80
    targetPort: 8080

Note that we're using `imagePullPolicy: Never` to ensure Kubernetes uses the locally built image rather than attempting to pull it from a registry.

Kubernetes services running with our deployed application ready for Telepresence interception

Setting Up the Local Environment for Debugging

To properly reproduce the production environment locally, we need to configure our local development environment with the same conditions. This means setting the ENV variable to "production" in our local run configuration.

Configuring the local development environment with production settings to replicate the issue

Using Telepresence to Intercept Kubernetes Traffic

Telepresence works by intercepting traffic meant for your Kubernetes service and redirecting it to your local development environment. This allows you to debug your application locally while it interacts with other services in the cluster.

Install Telepresence CLI if you haven't already
Connect to your Kubernetes cluster using Telepresence
Create an intercept for your service
Run your application locally
Set breakpoints in your code for debugging

When you connect with Telepresence, it creates a Traffic Manager pod in the ambassador namespace. This component routes and manages network traffic between your local environment and the Kubernetes cluster.

BASH

# Connect to the cluster with Telepresence
telepresence connect

# Create an intercept for the go-app service
telepresence intercept go-app --port 8080:8080

Debugging the Production Issue

With Telepresence intercepting traffic to our service, we can now debug the production issue locally. By setting breakpoints in our code and accessing the service through the Kubernetes endpoint, we can trace the execution and identify the root cause of the problem.

In our example, the issue occurs when the random number generated exceeds 5, causing an error in the data handler. Using a local debugger, we can inspect variables, step through code execution, and identify exactly what's happening when the error occurs.

// Breakpoint here to inspect the random number value
if os.Getenv("ENV") == "production" && randomNum > 5 {
    return errors.New("error loading data")
}

Benefits of Using Telepresence for Kubernetes Debugging

Using Telepresence for debugging Kubernetes pods offers several advantages over traditional debugging methods:

No need to access container shells or modify running pods
Full access to local debugging tools and IDEs
Real-time code changes without rebuilding containers
Testing against actual cluster dependencies and configurations
Faster development cycles and reduced time to fix issues

Conclusion: Streamlining Kubernetes Development

Telepresence bridges the gap between local development and Kubernetes environments, making it significantly easier to debug issues that only occur in production-like settings. By allowing developers to run code locally while connected to the cluster, it eliminates the time-consuming process of rebuilding and redeploying containers during development and debugging.

This approach not only speeds up the development cycle but also provides deeper insights into how applications behave in production environments, ultimately leading to more robust and reliable applications. For teams working with Kubernetes, Telepresence is an invaluable tool that should be part of the standard development workflow.

Streamline Kubernetes Pod Debugging with Telepresence: A Complete Developer Guide