Tailscale Kubernetes Operator Setup Guide

Overview

This guide walks you through setting up Tailscale Kubernetes Operator to expose your app (myapp.example.com) exclusively to your Tailscale network. After setup, only users connected to your Tailscale VPN will be able to access the app.

Architecture

Current Setup:

Internet → Cloudflare → NGINX Ingress → K8s Services (NextJS, FastAPI, RQ Dashboard)

After Tailscale Setup:

Tailscale Network → K8s Services (via Tailscale Operator)

Domain Options

Can I use my custom domain (myapp.example.com) with Tailscale?

Short answer: Yes, but not with a CNAME.

Why CNAME doesn’t work:

• MagicDNS names (e.g., myapp.your-tailnet.ts.net) only resolve within Tailscale network
• They don’t exist in public DNS
• A public CNAME myapp.example.com → myapp.your-tailnet.ts.net would fail to resolve

Your Options:

Recommended: Split-Horizon DNS

This guide focuses on Split-Horizon DNS because it:

• ✅ Lets you keep using myapp.example.com
• ✅ Works with free Tailscale plan
• ✅ Provides true VPN-only access
• ✅ Doesn’t require exit nodes or additional infrastructure

See Step 8.5 for detailed setup instructions.

Prerequisites

• Tailscale account (free tier works fine)
• Admin access to your GKE cluster
• kubectl configured for your cluster
• Helm installed locally
• Tailscale auth key (we’ll generate this in Step 1)

Step 1: Generate Tailscale Auth Key

1. Go to Tailscale Admin Console

2. Click Generate auth key

3. Configure the key:

   • Description: my-app-k8s-operator
   • Reusable: ✅ Enable (allows operator to create multiple nodes)
   • Ephemeral: ❌ Disable (keep nodes in your network permanently)
   • Pre-authorized: ✅ Enable (auto-approve devices)
   • Tags: Add tag tag:k8s (create if doesn’t exist)
   • Expiration: Set to 90 days or never expire

4. Copy the generated key (starts with tskey-auth-...)

5. Store it temporarily - you’ll need it in Step 3

Step 2: Install Tailscale Kubernetes Operator

Option A: Using Helm (Recommended)

# Add Tailscale Helm repository
helm repo add tailscale https://pkgs.tailscale.com/helmcharts
helm repo update

# Install the operator
helm upgrade --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  --set-string oauth.clientId="<YOUR_OAUTH_CLIENT_ID>" \
  --set-string oauth.clientSecret="<YOUR_OAUTH_CLIENT_SECRET>" \
  --wait

Note: For OAuth credentials, go to Tailscale OAuth Clients and create a new OAuth client with scopes:

• devices:write

Option B: Using kubectl (Alternative)

# Download and apply the operator manifest
kubectl apply -f https://github.com/tailscale/tailscale/raw/main/cmd/k8s-operator/deploy/manifests/operator.yaml

# Verify installation
kubectl get pods -n tailscale-system

Step 3: Create Tailscale OAuth Client (For Operator)

The operator needs OAuth credentials to manage devices on your behalf.

1. Go to Tailscale Admin Console → Settings → OAuth Clients

2. Click Generate OAuth client

3. Configure:

   • Description: k8s-operator-my-app
   • Scopes:
     • devices:write (required)
     • routes:write (optional, for subnet routing)
     • dns:write (optional, for MagicDNS)
   • Tags: tag:k8s

4. Copy the Client ID and Client Secret

5. Create Kubernetes secret:

kubectl create secret generic operator-oauth \
  --namespace=tailscale \
  --from-literal=client-id="<YOUR_OAUTH_CLIENT_ID>" \
  --from-literal=client-secret="<YOUR_OAUTH_CLIENT_SECRET>"

Step 4: Configure Operator to Use OAuth Secret

Create a values file for the operator:

cat > tailscale-operator-values.yaml <<EOF
oauth:
  clientId: "<YOUR_OAUTH_CLIENT_ID>"
  clientSecret: "<YOUR_OAUTH_CLIENT_SECRET>"

apiServerProxyConfig:
  mode: "true"
EOF

Update the operator installation:

helm upgrade --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  -f tailscale-operator-values.yaml \
  --wait

Step 5: Expose Services to Tailscale

You have two approaches:

Approach A: Service-Level Exposure (Recommended)

Expose individual services by adding annotations to each service.

Update your Helm chart templates

Edit k8s/charts/my-app/templates/service.yaml (or individual service files):

For NextJS Service:

apiVersion: v1
kind: Service
metadata:
  name: app-nextjs
  namespace: { { .Release.Namespace } }
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/hostname: "app-nextjs" # Will be accessible as app-nextjs.your-tailnet.ts.net
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: app-nextjs

For FastAPI Service:

apiVersion: v1
kind: Service
metadata:
  name: app-fastapi
  namespace: { { .Release.Namespace } }
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/hostname: "app-fastapi" # Will be accessible as app-fastapi.your-tailnet.ts.net
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  ports:
    - name: http
      port: 8000
      targetPort: 8000
  selector:
    app: app-fastapi

For RQ Dashboard Service:

apiVersion: v1
kind: Service
metadata:
  name: app-rqdash
  namespace: { { .Release.Namespace } }
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/hostname: "app-rqdash" # Will be accessible as app-rqdash.your-tailnet.ts.net
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  ports:
    - name: http
      port: 9181
      targetPort: 9181
  selector:
    app: app-rqdash

Approach B: Ingress-Level Exposure

Create a single Tailscale ingress proxy that routes to all your services.

Create k8s/charts/my-app/templates/tailscale-ingress.yaml:

{{- if .Values.tailscale.enabled -}}
apiVersion: v1
kind: Service
metadata:
  name: app-tailscale-ingress
  namespace: {{ .Release.Namespace }}
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/hostname: "myapp"  # Accessible as myapp.your-tailnet.ts.net
    tailscale.com/tags: "tag:k8s,tag:myapp"
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  ports:
    - name: http
      port: 80
      targetPort: 80
    - name: https
      port: 443
      targetPort: 443
  selector:
    app: app-nginx-proxy
---
# NGINX proxy to route to internal services based on path
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-nginx-proxy
  namespace: {{ .Release.Namespace }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-nginx-proxy
  template:
    metadata:
      labels:
        app: app-nginx-proxy
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      volumes:
      - name: nginx-config
        configMap:
          name: app-nginx-proxy-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-nginx-proxy-config
  namespace: {{ .Release.Namespace }}
data:
  nginx.conf: |
    events {
      worker_connections 1024;
    }
    http {
      upstream nextjs {
        server app-nextjs:3000;
      }
      upstream fastapi {
        server app-fastapi:8000;
      }
      upstream rqdash {
        server app-rq-dashboard:9181;
      }
      server {
        listen 80;
        server_name _;

        # FastAPI endpoints
        location /fastapi {
          rewrite ^/fastapi/(.*)$ /$1 break;
          proxy_pass http://fastapi;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
        }

        # RQ Dashboard
        location /rqdash {
          proxy_pass http://rqdash;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
        }

        # NextJS (default/catch-all)
        location / {
          proxy_pass http://nextjs;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
        }
      }
    }
{{- end }}

Add to your values.yaml:

tailscale:
  enabled: false # Set to true when ready to enable
  hostname: "myapp" # Will be accessible as myapp.your-tailnet.ts.net

Step 6: Deploy Updated Configuration

1. Update Helm chart version in k8s/charts/my-app/Chart.yaml:

   version: 0.2.0 # Bump version

2. Update values in values-orbit.yaml:

   tailscale:
     enabled: true
     hostname: "myapp"

3. Deploy:

   # If using local Helm
   helm upgrade my-app ./k8s/charts/my-app \
     -f path/to/values-orbit.yaml \
     --namespace my-app
   
   # Or commit and let your CI/CD pipeline deploy
   git add .
   git commit -m "feat: add Tailscale ingress support"
   git push

Step 7: Verify Tailscale Devices

1. Go to Tailscale Admin Console → Machines

2. You should see new devices:

   • app-nextjs (if using Approach A)
   • app-fastapi (if using Approach A)
   • app-rqdash (if using Approach A)
   • myapp (if using Approach B)

3. Each device will have a Tailscale IP (100.x.x.x) and MagicDNS name

Step 8: Access Your App via Tailscale

1. Connect to Tailscale on your device

2. Access the app using MagicDNS:

   If using Approach A (individual services):

   • NextJS: http://app-nextjs.your-tailnet.ts.net:3000
   • FastAPI: http://app-fastapi.your-tailnet.ts.net:8000
   • RQ Dashboard: http://app-rqdash.your-tailnet.ts.net:9181

   If using Approach B (single proxy):

   • Main app: http://myapp.your-tailnet.ts.net
   • FastAPI: http://myapp.your-tailnet.ts.net/fastapi
   • RQ Dashboard: http://myapp.your-tailnet.ts.net/rqdash

3. Test access - verify you can only access when connected to Tailscale

Step 8.5: Use Your Custom Domain (Split-Horizon DNS)

Optional but Recommended: Keep using myapp.example.com instead of MagicDNS hostnames.

Why Split-Horizon DNS?

• ✅ Users can continue using familiar domain myapp.example.com
• ✅ Works with free Tailscale plan
• ✅ Domain only resolves when connected to Tailscale
• ✅ Public DNS either doesn’t resolve or shows “VPN Required” page
• ❌ Cannot use CNAME to MagicDNS (MagicDNS names don’t exist in public DNS)

Setup Split-Horizon DNS

Part 1: Configure Tailscale DNS Override

1. Get your Tailscale service IP:

   Connect to Tailscale and check the IP assigned to your service:

   # If using Approach B (single proxy)
   tailscale status | grep "myapp "
   # Output: 100.x.x.x   myapp.your-tailnet.ts.net   tagged-devices
   
   # Or check Tailscale admin console
   # Go to Machines → find "myapp" → note the IP (100.x.x.x)

2. Add DNS override in Tailscale:

   Go to Tailscale Admin → DNS

   a. Scroll to “Search domains” section

   b. Click “Add search domain”

   c. Enter your domain: example.com

   d. Scroll to “Custom DNS” or “Split DNS” section

   e. Click “Add nameserver” → “Custom”

   f. Add an override record:

   Domain: myapp.example.com
   Value: 100.x.x.x  (your Tailscale service IP from step 1)

   Alternative method - Using Tailscale DNS config (if available):

   If your Tailscale version supports it, you can also configure via the ACL file:

   {
     "dns": {
       "nameservers": ["100.100.100.100"],
       "routes": {
         "example.com": ["100.100.100.100"]
       },
       "extraRecords": [
         {
           "name": "myapp.example.com",
           "type": "A",
           "value": "100.x.x.x"
         }
       ]
     }
   }

3. Verify Tailscale DNS override:

   On a device connected to Tailscale:

   # Test DNS resolution
   nslookup myapp.example.com
   # Should return 100.x.x.x (your Tailscale IP)
   
   # Test access
   curl http://myapp.example.com
   # Should load your app

Part 2: Configure Public DNS (Cloudflare)

You have two options for public DNS:

Option A: Remove public DNS record (Recommended)

1. Go to Cloudflare Dashboard
2. Select your domain example.com
3. Go to DNS → Records
4. Find the myapp A/CNAME record
5. Delete it or Pause it (disable the proxy)

Result: myapp.example.com won’t resolve publicly - only works for Tailscale users.

Important Cloudflare Note:

• If you keep the DNS record, ensure the Cloudflare proxy is disabled (grey cloud, not orange)
• When using Tailscale, you don’t need Cloudflare’s proxy features
• The orange cloud (proxied) can interfere with Tailscale’s DNS resolution

Option B: Point to “VPN Required” page

1. Create a simple static HTML page or deploy a placeholder service

2. Update the myapp DNS record to point to this placeholder:

   Type: A
   Name: myapp
   Content: <IP of your VPN-required page>
   Proxy status: Proxied (orange cloud)

3. Example “VPN Required” HTML:

   <!DOCTYPE html>
   <html>
     <head>
       <title>VPN Required</title>
       <style>
         body {
           font-family: Arial;
           text-align: center;
           padding: 50px;
         }
         .container {
           max-width: 600px;
           margin: 0 auto;
         }
       </style>
     </head>
     <body>
       <div class="container">
         <h1>🔒 VPN Required</h1>
         <p>This application requires connection to the company VPN.</p>
         <p>Please connect to Tailscale and try again.</p>
         <a href="https://tailscale.com/download">Download Tailscale</a>
       </div>
     </body>
   </html>

Result: Public users see a “VPN Required” message; Tailscale users access the real app.

Part 3: Test Split-Horizon DNS

1. Test from Tailscale-connected device:

   # Connect to Tailscale
   tailscale up
   
   # Test DNS
   nslookup myapp.example.com
   # Should return 100.x.x.x (Tailscale IP)
   
   # Test access
   curl -I http://myapp.example.com
   # Should return 200 OK from your app
   
   # Visit in browser
   open http://myapp.example.com

2. Test from non-Tailscale device:

   # Disconnect from Tailscale
   tailscale down
   
   # Test DNS
   nslookup myapp.example.com
   # Should either fail or return placeholder IP
   
   # Test access
   curl -I http://myapp.example.com
   # Should either fail or return VPN required page

Part 4: Update Your App Configuration (Optional)

If your app has hardcoded domain references, you may want to update environment variables:

In values-orbit.yaml:

ingress:
  enabled: false # Disable public ingress since we're using Tailscale
  host: "myapp.example.com" # Keep for reference

tailscale:
  enabled: true
  hostname: "myapp" # This creates the Tailscale device
  customDomain: "myapp.example.com" # Document the custom domain mapping

Troubleshooting Split-Horizon DNS

Issue: Domain still resolves to public IP when on Tailscale

Solution: DNS caching issue. Clear DNS cache:x`

# macOS
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

# Linux
sudo systemd-resolve --flush-caches

# Windows
ipconfig /flushdns

Issue: Domain doesn’t resolve at all

1. Verify MagicDNS is enabled:

   • Go to Tailscale Admin → DNS
   • Enable “MagicDNS”

2. Verify DNS override is correct:

   • Check that myapp.example.com maps to correct Tailscale IP
   • Confirm the Tailscale device is online

3. Test direct Tailscale IP:

   curl http://100.x.x.x
   # Should work if DNS override is the issue

Issue: Works from some devices but not others

• DNS overrides are tailnet-wide, so this is likely a DNS caching issue
• Try connecting/reconnecting to Tailscale on the problematic device
• Check DNS settings on the device (shouldn’t override Tailscale DNS)

Benefits of Split-Horizon DNS

✅ Familiar URLs: Team uses the same domain they’re used to ✅ No training needed: Users don’t need to remember MagicDNS hostnames ✅ Works with bookmarks: Existing bookmarks continue to work ✅ Free: No need for Tailscale Business plan ✅ True VPN-only access: Domain literally doesn’t resolve publicly ✅ Flexible: Can easily add more subdomains later

HTTPS Support for Custom Domain

If you want HTTPS on myapp.example.com:

Option 1: Tailscale HTTPS (Automatic)

Tailscale automatically provisions HTTPS certificates for MagicDNS names, but not for custom domains on free plans.

For myapp.example.com, you would need:

• Tailscale Business/Enterprise plan with custom domain support
• OR use one of the options below

Option 2: Let’s Encrypt with DNS-01 Challenge

Since the domain doesn’t resolve publicly, HTTP-01 challenge won’t work. Use DNS-01:

1. Install cert-manager in your cluster:

   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

2. Configure Cloudflare API token for cert-manager

3. Create certificate:

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: app-example-com-tls
     namespace: my-app
   spec:
     secretName: app-example-com-tls
     issuer: letsencrypt-prod
     dnsNames:
       - myapp.example.com
     solvers:
       - dns01:
           cloudflare:
             apiTokenSecretRef:
               name: cloudflare-api-token
               key: api-token

4. Update Tailscale service to use HTTPS:

   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       tailscale.com/expose: "true"
       tailscale.com/hostname: "myapp"
       tailscale.com/tls-secret: "app-example-com-tls" # Reference cert-manager secret

Option 3: Self-Signed Certificate (Development Only)

For testing/development, you can use a self-signed cert:

# Generate self-signed certificate
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt \
  -subj "/CN=myapp.example.com"

# Create Kubernetes secret
kubectl create secret tls app-example-com-tls \
  --cert=tls.crt --key=tls.key \
  -n my-app

Users will see a certificate warning but can proceed.

Option 4: Just Use HTTP (Recommended for Private VPN)

Since traffic is:

• Only accessible via Tailscale VPN
• Already encrypted by WireGuard
• Not exposed to public internet

HTTP is actually fine for internal use. You get:

• ✅ All traffic encrypted by Tailscale/WireGuard (stronger than TLS)
• ✅ No certificate management complexity
• ✅ Simpler troubleshooting
• ❌ Browser may show “Not Secure” (even though traffic IS encrypted)

Recommendation: Start with HTTP. Add HTTPS later if needed for compliance or browser compatibility.

Step 9: Configure Tailscale ACLs (Optional but Recommended)

Control who can access what with Tailscale ACLs.

1. Go to Tailscale Admin Console → Access Controls

2. Add ACL rules:

{
  "acls": [
    // Allow all users in your tailnet to access the app
    {
      "action": "accept",
      "src": ["autogroup:members"],
      "dst": ["tag:k8s:*"]
    },

    // Or restrict to specific users/groups
    {
      "action": "accept",
      "src": ["group:developers"],
      "dst": ["tag:k8s:*"]
    }
  ],

  "tagOwners": {
    "tag:k8s": ["autogroup:admin"]
  },

  "groups": {
    "group:developers": ["[email protected]", "[email protected]"]
  }
}

Step 10: Disable Public Ingress (Optional)

Once Tailscale is working, you can disable public access:

1. Update values-orbit.yaml:

   ingress:
     enabled: false # Disable public ingress
   
   tailscale:
     enabled: true # Keep Tailscale enabled

2. Or keep ingress enabled but restrict to your IP (as you already have):

   ingress:
     enabled: true
     annotations:
       nginx.ingress.kubernetes.io/whitelist-source-range: "203.0.113.10/32"

3. Redeploy:

   helm upgrade my-app ./k8s/charts/my-app \
     -f path/to/values-orbit.yaml \
     --namespace my-app

Step 11: Share Access with Team Members

1. Invite users to Tailscale:

   • Go to Tailscale Admin → Users
   • Click “Invite user”
   • Send invite link

2. Users install Tailscale:

   • Desktop: https://tailscale.com/download
   • Mobile: App Store / Play Store
   • Linux: curl -fsSL https://tailscale.com/install.sh | sh

3. Users connect and access:

   • Connect to Tailscale
   • Access http://myapp.your-tailnet.ts.net

Troubleshooting

Operator pods not starting

# Check operator logs
kubectl logs -n tailscale -l app=operator

# Check operator deployment
kubectl describe deployment -n tailscale tailscale-operator

Services not getting Tailscale IPs

# Check service status
kubectl get svc -n my-app

# Check service events
kubectl describe svc app-nextjs -n my-app

# Check operator logs for errors
kubectl logs -n tailscale -l app=operator --tail=100

Cannot access via MagicDNS

1. Verify MagicDNS is enabled:

   • Go to Tailscale Admin → DNS
   • Enable “MagicDNS”

2. Check your tailnet name:

   • Go to Tailscale Admin → Settings
   • Note your “Tailnet name” (e.g., example.ts.net)

3. Verify device is showing up:

   • Go to Tailscale Admin → Machines
   • Search for your service hostname

”Permission denied” or “Access denied”

Check your ACLs:

# Test access from your device
tailscale ping myapp.your-tailnet.ts.net

# Check which devices you can access
tailscale status

Port conflicts or routing issues

If using Approach A (individual services), ensure each service uses unique ports.

If using Approach B (proxy), check NGINX logs:

kubectl logs -n my-app deployment/app-nginx-proxy

Clean Up / Rollback

If you need to revert:

1. Disable Tailscale in values:

   tailscale:
     enabled: false

2. Re-enable public ingress:

   ingress:
     enabled: true

3. Redeploy:

   helm upgrade my-app ./k8s/charts/my-app \
     -f path/to/values-orbit.yaml \
     --namespace my-app

4. Remove Tailscale devices (optional):

   • Go to Tailscale Admin → Machines
   • Delete the K8s service devices

5. Uninstall operator (if completely removing):

   helm uninstall tailscale-operator -n tailscale
   kubectl delete namespace tailscale

Benefits Summary

✅ Security: App only accessible via Tailscale VPN ✅ Identity-based access: Use Tailscale’s user/group-based ACLs ✅ No public exposure: Remove from public internet entirely ✅ Easy sharing: Just invite users to Tailscale ✅ MagicDNS: Clean hostnames instead of IP addresses ✅ Zero-config networking: No firewall rules or port forwarding ✅ Encrypted traffic: All traffic encrypted by WireGuard

Next Steps

• Set up Tailscale ACLs to control team access
• Configure MagicDNS custom domain (optional)
• Set up Tailscale SSH for secure pod access
• Monitor Tailscale traffic in admin console
• Document access procedures for team

References

• Tailscale Kubernetes Operator Docs
• Tailscale ACL Reference
• Tailscale MagicDNS
• Tailscale Service Exposure