Development Guide

Dependencies

The following tools are required for contributing to the Karpenter project.

Developing

Setup / Teardown

Based on how you are running your Kubernetes cluster, follow the Environment specific setup to configure your environment before you continue. Once you have your environment set up, to install Karpenter in the Kubernetes cluster specified in your ~/.kube/config run the following commands.

CLOUD_PROVIDER="<YOUR_PROVIDER>" make apply # Install Karpenter
make delete # Uninstall Karpenter

Developer Loop

• Make sure dependencies are installed
  • Run make codegen to make sure yaml manifests are generated
  • Run make toolchain to install cli tools for building and testing the project
• You will need a personal development image repository (e.g. ECR)
  • Make sure you have valid credentials to your development repository.
  • $KO_DOCKER_REPO must point to your development repository
  • Your cluster must have permissions to read from the repository
• It’s also a good idea to persist ${CLOUD_PROVIDER} in your environment variables to simplify the make apply command.

Build and Deploy

Note: these commands do not rely on each other and may be executed independently

make apply # quickly deploy changes to your cluster
make dev # run codegen, lint, and tests

Testing

make test       # E2E correctness tests
make battletest # More rigorous tests run in CI environment

Change Log Level

kubectl patch configmap config-logging -n karpenter --patch '{"data":{"loglevel.controller":"debug"}}' # Debug Level
kubectl patch configmap config-logging -n karpenter --patch '{"data":{"loglevel.controller":"info"}}' # Info Level

Debugging Metrics

OSX:

open http://localhost:8080/metrics && kubectl port-forward service/karpenter -n karpenter 8080

Linux:

gio open http://localhost:8080/metrics && kubectl port-forward service/karpenter -n karpenter 8080

Tailing Logs

While you can tail Karpenter’s logs with kubectl, there’s a number of tools out there that enhance the experience. We recommend Stern:

stern -n karpenter -l app.kubernetes.io/name=karpenter

Environment specific setup

AWS

Set the CLOUD_PROVIDER environment variable to build cloud provider specific packages of Karpenter.

export CLOUD_PROVIDER="aws"

For local development on Karpenter you will need a Docker repo which can manage your images for Karpenter components. You can use the following command to provision an ECR repository.

aws ecr create-repository \
    --repository-name karpenter/controller \
    --image-scanning-configuration scanOnPush=true \
    --region "${AWS_DEFAULT_REGION}"
aws ecr create-repository \
    --repository-name karpenter/webhook \
    --image-scanning-configuration scanOnPush=true \
    --region "${AWS_DEFAULT_REGION}"

Once you have your ECR repository provisioned, configure your Docker daemon to authenticate with your newly created repository.

export KO_DOCKER_REPO="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/karpenter"
aws ecr get-login-password --region "${AWS_DEFAULT_REGION}" | docker login --username AWS --password-stdin "${KO_DOCKER_REPO}"

Profiling memory

Karpenter exposes a pprof endpoint on its metrics port.

Learn about profiling with pprof: https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/

Prerequisites

brew install graphviz
go install github.com/google/pprof@latest

Get a profile

# Connect to the metrics endpoint
kubectl port-forward service/karpenter -n karpenter 8080
open http://localhost:8080/debug/pprof/
# Visualize the memory
go tool pprof -http 0.0.0.0:9000 localhost:8080/debug/pprof/heap