ArgoCD Bootstrap Pattern Skill

Overview

ArgoCD bootstrap follows the "App-of-Apps" pattern: a root Application that creates other Applications. This enables GitOps-driven deployment of all cluster resources from a single Git repository.

Architecture

Terraform
   ↓
ArgoCD Helm Chart Installation
   ↓
Bootstrap Application (created by Terraform)
   ↓
ApplicationSet (created by Bootstrap)
   ↓
Individual Applications (created by ApplicationSet)
   ↓
Runner Scale Sets (managed by Applications)

Bootstrap Application Pattern

Level 1: Terraform Creates Bootstrap Application

Terraform creates a single "root" Application that points to the ArgoCD manifests directory:

resource "kubectl_manifest" "argocd_bootstrap" {
  yaml_body = yamlencode({
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"
    metadata = {
      name      = "bootstrap"
      namespace = "argocd"
    }
    spec = {
      project = "default"
      source = {
        repoURL        = "https://github.com/Matchpoint-AI/matchpoint-github-runners-helm"
        targetRevision = "main"
        path           = "argocd"  # Directory with ApplicationSets
      }
      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = "argocd"
      }
      syncPolicy = {
        automated = {
          prune    = true
          selfHeal = true
        }
      }
    }
  })

  depends_on = [
    helm_release.argocd,
    time_sleep.wait_for_crds
  ]
}

Key fields:

• path: "argocd" - Directory containing ApplicationSets and Applications
• targetRevision: "main" - Git branch to sync (use specific tag for production)
• automated.prune: true - Remove resources deleted from Git
• automated.selfHeal: true - Revert manual changes to match Git

Level 2: Bootstrap Creates ApplicationSet

The argocd/ directory contains ApplicationSet manifests:

# argocd/applicationset-runners.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: github-runners
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - name: arc-beta-runners
        namespace: arc-runners
        valuesFile: examples/beta-runners-values.yaml
      - name: arc-frontend-runners
        namespace: arc-frontend-runners
        valuesFile: examples/frontend-runners-values.yaml
  template:
    metadata:
      name: '{{name}}'
      namespace: argocd
      finalizers:
      - resources-finalizer.argocd.argoproj.io
    spec:
      project: default
      source:
        repoURL: https://github.com/Matchpoint-AI/matchpoint-github-runners-helm
        targetRevision: main
        path: charts/github-actions-runners
        helm:
          releaseName: '{{name}}'  # CRITICAL: Must match runnerScaleSetName
          valueFiles:
          - '../../{{valuesFile}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true
        - PruneLast=true

Key concepts:

Generators - Define list of Applications to create:

generators:
- list:
    elements:
    - name: arc-beta-runners      # Application name
      namespace: arc-runners       # Target namespace
      valuesFile: examples/beta.yaml  # Helm values file

Template - Define Application spec using generator variables:

template:
  metadata:
    name: '{{name}}'  # Substituted from generator

Helm configuration:

helm:
  releaseName: '{{name}}'  # MUST match runnerScaleSetName in values!
  valueFiles:
  - '../../{{valuesFile}}'  # Path relative to chart directory

Level 3: ApplicationSet Creates Applications

The ApplicationSet generates individual Applications for each runner pool:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: arc-beta-runners
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/Matchpoint-AI/matchpoint-github-runners-helm
    targetRevision: main
    path: charts/github-actions-runners
    helm:
      releaseName: arc-beta-runners
      valueFiles:
      - ../../examples/beta-runners-values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: arc-runners
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Critical Configuration: releaseName vs runnerScaleSetName

The Mismatch Problem

Issue #112 Root Cause: ApplicationSet releaseName didn't match Helm chart runnerScaleSetName:

# ApplicationSet
releaseName: arc-runners  # ❌ Kubernetes resource name

# examples/runners-values.yaml
runnerScaleSetName: "arc-beta-runners"  # ✅ GitHub label

Impact:

• ArgoCD tracks resources under arc-runners
• AutoScalingRunnerSet created with name arc-beta-runners
• Resource tracking fails → stale runners with empty labels

The Fix

BOTH must match:

# argocd/applicationset-runners.yaml
generators:
- list:
    elements:
    - name: arc-beta-runners  # Used for releaseName

template:
  spec:
    source:
      helm:
        releaseName: '{{name}}'  # = arc-beta-runners

# examples/beta-runners-values.yaml
gha-runner-scale-set:
  runnerScaleSetName: "arc-beta-runners"  # MUST MATCH releaseName

Workflow must also match:

# .github/workflows/ci.yaml
jobs:
  build:
    runs-on: arc-beta-runners  # MUST MATCH runnerScaleSetName

ApplicationSet Features

1. List Generator

Static list of elements:

generators:
- list:
    elements:
    - name: app1
      env: prod
    - name: app2
      env: staging

2. Git Generator

Generate from files in Git repository:

generators:
- git:
    repoURL: https://github.com/org/repo
    revision: main
    files:
    - path: "apps/*/config.yaml"

3. Matrix Generator

Combine multiple generators:

generators:
- matrix:
    generators:
    - list:
        elements:
        - cluster: prod
        - cluster: staging
    - list:
        elements:
        - app: frontend
        - app: backend
# Creates: prod-frontend, prod-backend, staging-frontend, staging-backend

Sync Policies

Automated Sync

syncPolicy:
  automated:
    prune: true      # Delete resources removed from Git
    selfHeal: true   # Revert manual kubectl changes

Use when:

• Production with confidence in CI/CD
• Infrastructure as Code discipline enforced
• Changes only via Git

Manual Sync

syncPolicy: {}  # No automated sync

Use when:

• Testing/staging environments
• Manual approval required for changes
• Debugging configuration issues

Sync Options

syncOptions:
- CreateNamespace=true  # Create destination namespace if missing
- PruneLast=true        # Delete old resources after new ones ready
- Replace=true          # Use kubectl replace instead of apply

Directory Structure

matchpoint-github-runners-helm/
├── argocd/
│   ├── applicationset-runners.yaml   # Creates runner Applications
│   ├── applications/                  # Individual Applications (legacy)
│   │   └── arc-controller.yaml       # ARC controller Application
│   └── apps-live/                     # Alternative app definitions (not used)
├── charts/
│   └── github-actions-runners/       # Helm chart deployed by Applications
└── examples/
    ├── beta-runners-values.yaml      # Values for arc-beta-runners
    └── frontend-runners-values.yaml  # Values for arc-frontend-runners

Active paths:

• argocd/applicationset-runners.yaml - Generates runner Applications
• argocd/applications/arc-controller.yaml - Deploys ARC controller

Inactive paths:

• argocd/apps-live/ - Old structure, not referenced by bootstrap

Common Patterns

Pattern 1: Single Application (ARC Controller)

For resources that don't need templating:

# argocd/applications/arc-controller.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: arc-controller
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/Matchpoint-AI/matchpoint-github-runners-helm
    targetRevision: main
    path: charts/github-actions-controller
  destination:
    server: https://kubernetes.default.svc
    namespace: arc-systems
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Pattern 2: ApplicationSet (Runner Pools)

For resources that need multiple instances with different configurations:

# argocd/applicationset-runners.yaml
# (See full example above)

Troubleshooting

Error: "Application CRD not found"

Symptom:

error: unable to recognize "argocd/bootstrap.yaml": no matches for kind "Application"

Cause: ArgoCD CRDs not registered yet

Fix:

resource "time_sleep" "wait_for_crds" {
  depends_on      = [helm_release.argocd]
  create_duration = "30s"
}

Error: "Application OutOfSync"

Symptom:

kubectl get application arc-beta-runners -n argocd
# Status: OutOfSync

Diagnosis:

kubectl describe application arc-beta-runners -n argocd
# Check status.conditions for error details

Common causes:

1. targetRevision branch doesn't exist
2. path directory not found in repo
3. valueFiles path incorrect
4. Helm chart rendering failed

Fix:

# Force refresh
kubectl annotate application arc-beta-runners -n argocd \
  argocd.argoproj.io/refresh="hard" --overwrite

# Manual sync
kubectl patch application arc-beta-runners -n argocd \
  -p '{"operation":{"initiatedBy":{"automated":true}}}' --type=merge

Error: "Helm template failed"

Symptom:

ComparisonError: failed to render manifests: helm template failed

Diagnosis:

# Get error details
kubectl get application arc-beta-runners -n argocd -o yaml | grep -A 10 conditions

# Test helm template locally
helm template arc-beta-runners ./charts/github-actions-runners \
  -f examples/beta-runners-values.yaml \
  --namespace arc-runners

Common causes:

1. Invalid YAML in values file
2. Missing required values
3. Helm chart syntax error

Sync Stuck: "PruneLast" Waiting

Symptom: Application stuck syncing, old resources not deleted

Cause: PruneLast=true waits for new resources to be Ready before deleting old ones

Fix:

# Check pod status
kubectl get pods -n arc-runners

# If pods stuck, check events
kubectl get events -n arc-runners --sort-by='.lastTimestamp' | tail -20

# If safe to force delete old resources
kubectl delete application arc-beta-runners -n argocd
kubectl apply -f argocd/applicationset-runners.yaml

Diagnostic Commands

# List all Applications
kubectl get applications -n argocd

# List all ApplicationSets
kubectl get applicationset -n argocd

# Check Application status
kubectl describe application arc-beta-runners -n argocd

# View Application manifest
kubectl get application arc-beta-runners -n argocd -o yaml

# Check ApplicationSet generators
kubectl get applicationset github-runners -n argocd -o yaml | grep -A 20 generators

# View ArgoCD controller logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100

# Force Application refresh
kubectl annotate application arc-beta-runners -n argocd argocd.argoproj.io/refresh="hard" --overwrite

Best Practices

1. Use ApplicationSet for similar apps - Runner pools, microservices
2. Use individual Applications for unique apps - ARC controller, databases
3. Always set targetRevision - Prevents unexpected updates from main
4. Enable automated sync in production - Ensures Git is source of truth
5. Use PruneLast for stateful apps - Prevents downtime during updates
6. Match releaseName and runnerScaleSetName - Critical for ARC runners
7. Keep bootstrap simple - One root Application pointing to ArgoCD directory

Related Skills

• [arc-terraform-deployment](../arc-terraform-deployment/SKILL.md) - Terraform patterns for ArgoCD
• [infrastructure-cd](../infrastructure-cd/SKILL.md) - CI/CD workflows
• [arc-runner-troubleshooting](../arc-runner-troubleshooting/SKILL.md) - Runner issues

Related Issues

• #121 - releaseName/runnerScaleSetName mismatch
• #122 - ApplicationSet parameters fix
• #112 - Empty labels investigation

References

• ArgoCD App-of-Apps
• ApplicationSet Documentation
• ArgoCD Sync Options