{"id":1299,"date":"2026-02-13T09:30:18","date_gmt":"2026-02-13T08:30:18","guid":{"rendered":"https:\/\/simon-frey.com\/blog\/?p=1299"},"modified":"2026-03-10T14:24:20","modified_gmt":"2026-03-10T13:24:20","slug":"kubernetes-on-hetzner-cloud","status":"publish","type":"post","link":"https:\/\/simon-frey.com\/blog\/kubernetes-on-hetzner-cloud\/","title":{"rendered":"Production-ready Kubernetes homelab on Hetzner Cloud for ~\u20ac10\/month"},"content":{"rendered":"\n

I wanted a cheap, production-grade homelab Kubernetes cluster. Most cloud providers charge \u20ac70+\/month just for the control plane and my setup on Hetzner Cloud<\/a> runs for about \u20ac14\/month using Talos Linux<\/strong>, Cilium<\/strong>, and WireGuard<\/strong>. Everything is automated with Terraform, Packer, and ArgoCD<\/strong> for GitOps.<\/p>\n\n\n\n

As this is my ongoing, evolving homelab setup, this post might change from time to time. The Terraform code is split across multiple files (main.tf<\/code>, talos.tf<\/code>, argocd.tf<\/code>, etc.) and all workloads are managed declaratively through ArgoCD. The cluster runs worker nodes in multiple European zones, just for trying it out, no real valuable reason for a homesetup. With my small homelab cluster this has no real-world benefit, rather downsides due to cross-zone networking.<\/p>\n\n\n\n

This article is not meant as full 0-to-100 documentation (that is better done in the repo itself). Instead, I explain the design decisions and why I went for certain parts in the setup. The full code (Terraform, Packer, GitOps manifests) is available on GitHub: hetzner_k8s<\/code><\/a><\/p>\n\n\n\n

\n\n\n\n

Architecture overview<\/h2>\n\n\n\n

I decided to encapsulate the entire cluster management in a WireGuard harness, as this exposes the cluster to less risk of being hacked. Open to the world, I only have ports 80 (HTTP) and 443 (HTTPS), which are exposed via a Traefik load balancer (with certificates managed via Let’s Encrypt).<\/p>\n\n\n\n

This also secures the cluster-internal services (without an ingress). E.g. you can only access the monitoring stack (Grafana) or the ArgoCD UI via WireGuard tunnel + port forwarding. Might be a bit paranoid, but as I don’t want people running crypto miners on my neck (especially since the cluster has auto-scaling of nodes), better safe than sorry.<\/p>\n\n\n\n

The control plane node is tainted with NoSchedule<\/code>, so only infrastructure services (Cilium, Traefik, ArgoCD, CCM, CSI, metrics-server, monitoring) that carry explicit tolerations run there. Application workloads become Pending until the cluster autoscaler provisions worker nodes. This keeps the control plane stable while still allowing the cluster to function without any workers for pure infrastructure.<\/p>\n\n\n\n

Building the Talos image with Packer<\/h2>\n\n\n\n

Hetzner doesn’t offer Talos Linux as a native image. You build a snapshot using Packer and Hetzner’s rescue mode, a temporary Linux environment that lets you write directly to the server’s disk.<\/p>\n\n\n\n

The Packer config lives in hetzner_k8s\/packer\/hcloud.pkr.hcl<\/code>:<\/p>\n\n\n\n

source \"hcloud\" \"talos\" {\n  token       = var.hcloud_token\n  location    = var.server_location\n  server_type = var.server_type\n  server_name = \"packer-talos-builder\"\n\n  rescue      = \"linux64\"        # Boot into rescue mode\n  image       = \"ubuntu-24.04\"   # Base image (irrelevant, rescue overwrites it)\n\n  snapshot_name = \"talos-${var.talos_version}\"\n  snapshot_labels = {\n    os      = \"talos\"\n    version = var.talos_version\n  }\n\n  ssh_username = \"root\"\n}<\/code><\/pre>\n\n\n\n
The schematic ID (ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515<\/code>) is generated at factory.talos.dev<\/a> and encodes the qemu-guest-agent extension. Packer creates a snapshot that Terraform references by label:<\/p>\n\n\n\n
data \"hcloud_image\" \"talos\" {\n  with_selector     = \"os=talos\"\n  most_recent       = true\n  with_architecture = \"x86\"\n}<\/code><\/pre>\n\n\n\n
\n\n\n\n
Networking<\/h2>\n\n\n\n
A single private network connects all nodes. Hetzner’s eu-central zone spans three datacenters (fsn1, nbg1, hel1), so workers can be distributed for availability.<\/p>\n\n\n\n
Firewall rules<\/h3>\n\n\n\n
The firewall only allows HTTP\/HTTPS (for ingress), WireGuard (UDP 51820), and ICMP. There is no SSH as Talos doesn’t have an SSH server. Hetzner firewalls don’t filter private network traffic, so all inter-node communication is unrestricted by design.<\/p>\n\n\n\n
Load balancer<\/h3>\n\n\n\n
A Terraform-created lb11 sits on the private network at 10.0.1.254<\/code>. Its targets and service definitions are managed by the Hetzner CCM (not Terraform). The CCM watches for Kubernetes Services with matching annotations and auto-configures the LB.<\/p>\n\n\n\n
\n\n\n
\n  
\n  
Highly skilled DevOps\/SRE Freelancer<\/h3>\n  
I am Simon, the author of this blog. And I have great news: You can work with me<\/b><\/p>\n  
As DevOps and Infrastructure freelancer, I will help you choose the right Infrastructure technology for your company, fix your cloud problems and support your team in building scalable products.<\/p>\n  
I work with Golang, Docker, Kubernetes, Google Cloud, AWS and Terraform.<\/p>\n  
Checkout my CV<\/a> to learn more or directly contact me via the button below.<\/p>\n  <\/div>  \n  \"Simon\n\n  <\/div>\n  \n  Let’s work together!<\/a>\n<\/div><\/div>\n\n\n
Talos machine configuration<\/h2>\n\n\n\n
After first setting up the cluster with k3s on Ubuntu (Hetzner base image), I was looking for a different, more sophisticated solution and remembered Talos. Talos has the benefit of being immutable and more locked down, which helps getting the cluster into a reproducible, stable, and secure state.<\/p>\n\n\n\n
Here are the key design decisions baked into the control plane config (hetzner_k8s\/talos.tf<\/code>):<\/p>\n\n\n\n
CNI and kube-proxy disabled<\/strong>, Cilium takes over:<\/p>\n\n\n\n
cluster:\n  network:\n    cni:\n      name: none\n  proxy:\n    disabled: true<\/code><\/pre>\n\n\n\n
Control plane scheduling disabled<\/strong>, the node gets a NoSchedule<\/code> taint so only infrastructure pods with explicit tolerations run there:<\/p>\n\n\n\n
cluster:\n  allowSchedulingOnControlPlanes: false<\/code><\/pre>\n\n\n\n
WireGuard interface<\/strong> on the control plane, so you can reach the K8s API without exposing port 6443 publicly:<\/p>\n\n\n\n
machine:\n  network:\n    interfaces:\n      - interface: wg0\n        mtu: 1420\n        addresses: [\"10.200.200.1\/24\"]\n        wireguard:\n          privateKey: ${wireguard_server_private_key}\n          listenPort: 51820\n          peers:\n            - publicKey: ${wireguard_client_public_key}\n              allowedIPs: [\"10.200.200.2\/32\"]<\/code><\/pre>\n\n\n\n
etcd only advertised on private network<\/strong>, prevents etcd from binding to public IPs:<\/p>\n\n\n\n
cluster:\n  etcd:\n    advertisedSubnets: [\"10.0.1.0\/24\"]<\/code><\/pre>\n\n\n\n
\n\n\n\n
Secure access with WireGuard<\/h2>\n\n\n\n
As already mentioned in the intro, this might be a bit paranoid, but I just want to lock down the server as much as possible to not have to deal with any malicious actors. I go for a defense-in-depth approach of locking down everything that I don’t need publicly, and the control plane is one important part here. Hence the Kubernetes API (port 6443) and Talos API (port 50000) are never exposed publicly. Instead, I set up a WireGuard tunnel using wireproxy<\/strong>, a userspace WireGuard client that requires no root privileges and no kernel module. (Why wireproxy? I want to be able to set this up with a single command without any “sudo” access on my local machine. wg-quick needs kernel\/sudo access. Wireproxy keeps it all in userspace. And as the traffic I send over the tunnel is negligibly small, I don’t see a relevant performance downside.)<\/p>\n\n\n\n
Terraform generates a .wireproxy.conf<\/code> and starts it automatically:<\/p>\n\n\n\n
[Interface]\nPrivateKey = <client-private-key>\nAddress = 10.200.200.2\/32\nMTU = 1420\n\n[Peer]\nPublicKey = <server-public-key>\nEndpoint = <control-plane-public-ip>:51820\nAllowedIPs = 10.200.200.0\/24, 10.0.0.0\/16\nPersistentKeepalive = 25\n\n[TCPClientTunnel]\nBindAddress = 127.0.0.1:50000\nTarget = 10.200.200.1:50000\n\n[TCPClientTunnel]\nBindAddress = 127.0.0.1:6443\nTarget = 10.200.200.1:6443<\/code><\/pre>\n\n\n\n
After terraform apply<\/code>, kubectl<\/code> and talosctl<\/code> talk to 127.0.0.1<\/code>. Wireproxy tunnels the traffic through WireGuard to the control plane’s private VPN address. (For later connections to the cluster, you need to set up wireproxy again, as documented in the repo’s README.)<\/p>\n\n\n\n
\n\n\n\n
GitOps with ArgoCD<\/h2>\n\n\n\n
The cluster originally used a 686-line shell script (post-deploy.sh<\/code>) that sequentially installed every component via Helm and kubectl. That approach was fragile: no retries on transient failures, no drift detection, no self-healing, and it required manual re-runs whenever something changed. I migrated to ArgoCD<\/strong> using the app-of-apps pattern, and it was one of the best decisions for this project.<\/p>\n\n\n\n
What Terraform bootstraps<\/h3>\n\n\n\n
Terraform installs only the bare minimum that must exist before ArgoCD can function:<\/p>\n\n\n\n
    \n
  1. Cilium<\/strong> (Helm release): Without the CNI, nodes stay NotReady<\/code> and nothing can schedule.<\/li>\n\n\n\n
  2. Wait for CoreDNS<\/strong>: A null_resource<\/code> polls CoreDNS readiness so image pulls from external registries work.<\/li>\n\n\n\n
  3. ArgoCD<\/strong> (Helm release): Installed with tolerations for both the control-plane<\/code> and cloud-provider-uninitialized<\/code> taints, solving the chicken-and-egg problem where the CCM (which removes the cloud-provider taint) is itself deployed by ArgoCD.<\/li>\n\n\n\n
  4. Secrets and ConfigMaps<\/strong>: Terraform generates passwords (random_password<\/code>), SSH keys, and cloud-init configs and creates Kubernetes secrets that ArgoCD-managed apps reference by name. This keeps secrets out of git while letting ArgoCD manage the workloads.<\/li>\n\n\n\n
  5. Root Application<\/strong>: A single ArgoCD Application that points to the gitops\/root-app\/<\/code> Helm chart.<\/li>\n<\/ol>\n\n\n\n
    App-of-apps and sync waves<\/h3>\n\n\n\n
    The root Helm chart contains one ArgoCD Application template per component. Sync wave annotations control the installation order, just like the old shell script did, but with automatic retries, self-healing, and drift detection:<\/p>\n\n\n\n
    Wave<\/th>Component<\/th>Purpose<\/th><\/tr><\/thead>
    -5<\/td>Hetzner CCM<\/td>Removes cloud-provider taint, enables LB management<\/td><\/tr>
    -4<\/td>Hetzner CSI<\/td>Persistent volumes via Hetzner block storage<\/td><\/tr>
    -3<\/td>metrics-server<\/td>CPU\/memory metrics for HPA<\/td><\/tr>
    -2<\/td>Traefik<\/td>Ingress controller, auto-registered with Hetzner LB<\/td><\/tr>
    -1<\/td>cert-manager<\/td>TLS certificate automation<\/td><\/tr>
    0<\/td>cert-issuers<\/td>Let’s Encrypt staging + production ClusterIssuers<\/td><\/tr>
    4<\/td>Cluster autoscaler<\/td>Scales workers 0 to 5 based on pending pods<\/td><\/tr>
    5<\/td>Monitoring<\/td>VictoriaMetrics + Grafana (optional)<\/td><\/tr>
    5<\/td>kwatch<\/td>Cluster event notifications (optional)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    All applications share a common sync policy defined in a Helm helper (gitops\/root-app\/templates\/_helpers.tpl<\/code>):<\/p>\n\n\n\n
    {{- define \"root-app.syncPolicy\" -}}\nsyncPolicy:\n  automated:\n    prune: true\n    selfHeal: true\n  syncOptions:\n    - ServerSideApply=true\n  retry:\n    limit: 5\n    backoff:\n      duration: 5s\n      factor: 2\n      maxDuration: 3m\n{{- end -}}<\/code><\/pre>\n\n\n\n
    Server-side apply<\/h3>\n\n\n\n
    The ServerSideApply=true<\/code> setting deserves an explanation. By default, ArgoCD uses client-side apply, which stores the entire previous manifest in a kubectl.kubernetes.io\/last-applied-configuration<\/code> annotation on every resource. That annotation has a hard limit of 262144 bytes. Several CRDs in this cluster blow past that limit, particularly the six prometheus-operator CRDs shipped by the VictoriaMetrics Helm chart. When client-side apply hits the limit, the sync fails permanently and no amount of retries will fix it.<\/p>\n\n\n\n
    Server-side apply sidesteps this entirely. Instead of stuffing the previous state into an annotation, Kubernetes tracks field ownership via managedFields<\/code> in the resource metadata. There is no size limit. The API server also handles the diff and merge logic itself, which gives more accurate dry runs (including admission webhook validation) and proper conflict detection when multiple controllers manage the same resource. For the monitoring app specifically, ServerSideDiff=true<\/code> is also enabled so ArgoCD delegates the diff calculation to the API server too, which handles unknown status fields in CRDs more gracefully than client-side structured merge diff.<\/p>\n\n\n\n
    I initially only enabled ServerSideApply for the apps that needed it (KubeVirt, CDI, monitoring), but after hitting the annotation limit on yet another CRD, I moved it into the shared helper so every app gets it by default. There is no downside for smaller resources, and it prevents future surprises.<\/p>\n\n\n\n
    What post-deploy.sh does now<\/h3>\n\n\n\n
    The script went from 686 lines to about 75. All it does is fetch the kubeconfig from Terraform output, rewrite the API server address for the WireGuard tunnel, and print the cluster status with ArgoCD application sync states. No more Helm installs, no more kubectl apply, no more manual steps.<\/p>\n\n\n\n
    \n\n\n\n
    Ingress with Traefik and Hetzner LB<\/h2>\n\n\n\n
    Traefik runs as a DaemonSet (not a Deployment) so it’s present on every node, including the control plane. This is important because the control plane might be the only node when no workers are running. The Traefik values live in the ArgoCD Application template at gitops\/root-app\/templates\/traefik.yaml<\/code>:<\/p>\n\n\n\n
    deployment:\n  kind: DaemonSet\n\ntolerations:\n  - key: node-role.kubernetes.io\/control-plane\n    operator: Exists\n    effect: NoSchedule<\/code><\/pre>\n\n\n\n
    The Hetzner LB is configured via annotations on the Traefik Service. The CCM watches these and auto-manages the load balancer:<\/p>\n\n\n\n
    service:\n  annotations:\n    load-balancer.hetzner.cloud\/name: \"hetzner-k8s-lb\"\n    load-balancer.hetzner.cloud\/network-zone: eu-central\n    load-balancer.hetzner.cloud\/use-private-ip: \"true\"\n    load-balancer.hetzner.cloud\/uses-proxyprotocol: \"true\"<\/code><\/pre>\n\n\n\n
    Proxy protocol is essential. Without it, your application logs show the LB’s IP instead of the real client. Both Traefik and the LB must agree on proxy protocol being enabled:<\/p>\n\n\n\n
    ports:\n  web:\n    port: 80\n    proxyProtocol:\n      trustedIPs: [\"10.0.0.0\/8\", \"127.0.0.1\/32\"]\n    forwardedHeaders:\n      trustedIPs: [\"10.0.0.0\/8\", \"127.0.0.1\/32\"]\n  websecure:\n    port: 443\n    proxyProtocol:\n      trustedIPs: [\"10.0.0.0\/8\", \"127.0.0.1\/32\"]<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    Automatic TLS with cert-manager<\/h2>\n\n\n\n
    cert-manager handles Let’s Encrypt certificates automatically. Two ClusterIssuers are configured in gitops\/apps\/cert-issuers\/<\/code>: staging (for testing without rate limits) and production.<\/p>\n\n\n\n
    To get a certificate for your service, add a single annotation and a tls<\/code> block to your Ingress:<\/p>\n\n\n\n
    apiVersion: networking.k8s.io\/v1\nkind: Ingress\nmetadata:\n  name: hello\n  annotations:\n    cert-manager.io\/cluster-issuer: letsencrypt-prod\nspec:\n  ingressClassName: traefik\n  tls:\n    - hosts: [\"hello.k8s.example.com\"]\n      secretName: hello-tls\n  rules:\n    - host: hello.k8s.example.com\n      http:\n        paths:\n          - path: \/\n            pathType: Prefix\n            backend:\n              service:\n                name: hello\n                port:\n                  number: 80<\/code><\/pre>\n\n\n\n
    cert-manager sees the annotation, creates a Certificate resource, solves the HTTP-01 challenge through Traefik, and stores the signed certificate in the hello-tls<\/code> Secret. Renewal is automatic.<\/p>\n\n\n\n
    \n\n\n\n
    Cluster autoscaler<\/h2>\n\n\n\n
    The autoscaler uses the Hetzner cloud provider to spin up and tear down cx33 workers based on pending pod demand. The configuration in gitops\/apps\/autoscaler\/cluster-autoscaler.yaml<\/code> defines two node groups, one per datacenter location:<\/p>\n\n\n\n
    command:\n  - .\/cluster-autoscaler\n  - --cloud-provider=hetzner\n  - --nodes=0:5:cx33:nbg1:workers\n  - --nodes=0:5:cx33:hel1:workers\n  - --scale-down-enabled=true\n  - --scale-down-delay-after-add=5m\n  - --scale-down-unneeded-time=5m\n  - --scale-down-utilization-threshold=0.5\n  - --balance-similar-node-groups=true\n  - --expander=least-waste<\/code><\/pre>\n\n\n\n
    Each location can scale from 0 to 5 workers independently. balance-similar-node-groups<\/code> distributes workers evenly across nbg1 and hel1 for availability. least-waste<\/code> picks the group that would use resources most efficiently.<\/p>\n\n\n\n
    Terraform creates zero static workers (initial_worker_count = 0<\/code>). The autoscaler fully manages the worker lifecycle, spinning nodes up when pods are pending and tearing them down after 5 minutes of low utilization. This was a deliberate change from the earlier setup where Terraform created 2 permanent workers that the autoscaler couldn’t scale down without causing Terraform state drift.<\/p>\n\n\n\n
    The Talos worker machine config is passed to the autoscaler as a base64-encoded cloud-init secret. Terraform creates this secret directly in argocd.tf<\/code>, so there are no manual kubectl commands involved:<\/p>\n\n\n\n
    resource \"kubernetes_secret\" \"autoscaler\" {\n  metadata {\n    name      = \"hcloud-autoscaler\"\n    namespace = \"cluster-autoscaler\"\n  }\n  data = {\n    token      = var.hcloud_token\n    cloud-init = base64encode(data.talos_machine_configuration.worker.machine_configuration)\n    network    = hcloud_network.cluster.name\n    firewall   = hcloud_firewall.cluster.name\n    image      = data.hcloud_image.talos.id\n  }\n}<\/code><\/pre>\n\n\n\n
    The autoscaler deployment itself tolerates the control plane taint and prefers scheduling on the control plane, so it runs without any workers:<\/p>\n\n\n\n
    tolerations:\n  - key: node-role.kubernetes.io\/control-plane\n    operator: Exists\n    effect: NoSchedule\naffinity:\n  nodeAffinity:\n    preferredDuringSchedulingIgnoredDuringExecution:\n      - weight: 100\n        preference:\n          matchExpressions:\n            - key: node-role.kubernetes.io\/control-plane\n              operator: Exists<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    Monitoring with VictoriaMetrics<\/h2>\n\n\n\n
    The cluster originally ran Prometheus via kube-prometheus-stack, but I switched to VictoriaMetrics<\/strong> for lower resource usage. On a small cx33 node with 8 GB RAM, every megabyte counts, and VictoriaMetrics delivers the same functionality with a noticeably smaller footprint.<\/p>\n\n\n\n
    The monitoring stack is deployed as an optional ArgoCD Application (gated behind enable_monitoring<\/code>) using the victoria-metrics-k8s-stack<\/code> Helm chart (v0.72.4). It includes:<\/p>\n\n\n\n