• you start your project
• put everything in 1 directory
• maybe split files by broader domains.

.
├── databases.tf
├── vpc.tf
├── main.tf
├── outputs.tf
└── terraform.tf

• oh well, you need a staging environment
• both environments are very much the same
• you refactor the code to be parameterised by variables
• you provide 2 .tfvars files

.
├── production.tfvars
├── staging.tfvars
├── databases.tf
├── vpc.tf
├── variables.tf
├── main.tf
└── terraform.tf

• you add more services and they need infra
• the infra is similar
• you want to keep the code DRY³
• you create a repo, codify best practices, tag them for versioning
• you pull in modules via git

# select a specific tag
module "rds" {
  source = "github.com/example/rds?ref=v1.2.0"
}

• as the stack grows, the environments differ
• you start separating the code in larger blocks
  • the base environment
  • the services
• the code is pulled in as modules
• the services module receives output of base as input eg. vpc_id or subnets
• terraform apply plan is run manually still

.
├── environments
│   ├── production
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   └── staging
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
└── modules
    ├── base
    │   ├── main.tf
    │   ├── outputs.tf
    │   ├── variables.tf
    │   └── vpc.tf
    └── services
        ├── databases.tf
        ├── main.tf
        ├── outputs.tf
        └── variables.tf

The situation

• as the stack grows further, the amount of resources does as well
• each run of terraform plan -out plan takes more and more time
• to review and apply changes for developers becomes a dayfilling job
• you start cheating by targeted apply
• you notice that the amount of files downloaded for each terraform step is enormous⁴
• you notice that git tags can not be used for semantic versioning (version)

Possible solutions

• to address the versioning and data transfer issues - use a private Terraform module registry
• to address the runtime and ownership issue - split the stacks and let the teams handle them (DevOps style)

Don’t

• separate services along team borders⁵ ➡️ teams and responsibilities change, always
  
• share states between services ➡️ there are secrets in there!⁶ ➡️ read the docs of the terraform_remote_state data source!

Do

• Layer your stacks - account, network, clusters and services
• 1 Terraform stack per service
  • good for least privilige access
  • place the Terraform code into the service repo
• run the TF stacks in automation
• use an indirect way to share information between stacks⁷

• use structured data ➡️ ideally JSON for jsondecode() and jsonencode()
• use whatever storage you prefer ➡️ SSM Parameter Store or S3

Code for 3 Terraform modules will be provided

• s3_json_store CRUD JSON data on S3
• ssm_json_store CRUD JSON data on SSM Parameter store
• ssm_json_regex read SSM parameter with regex

• Data contracts between stacks
  • dependencies
  • versioning
• Dependencies of stacks
  • TF and Service code must be able to handle missing dependencies
  • reconciliation of TF stacks to check changed upstreams
  • eventually consistent

• Stack orchestration
  • state management should be centralised
  • stack execution should be in automation
• Permission management
  • for code changes (eg. CODEOWNERS)
  • for infrastructure changes
  • for accessing resources

What is GitOps?

GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. – https://about.gitlab.com/topics/gitops/

• In our context - pull vs. push principle
  • You don’t care in which environment a stack runs in
  • They are ready for your stack and your code is pulled in (vs. pushed via a pipeline)

• Each environment must be configurable individually
  • has its own entry point for FluxCD
  • this allows for configuration of deployed services
• For audit reasons, production environments must use fixed service versions, others can use semantic versioning
  
• Flux applications must be DRY
  • do not c&p code
  • implication: no individual configuration of apps

• in the central Flux repo there are NO variables, parameters etc. pp.
  • we only document the intent to run a service
  • self-configuration happens inside of an environment
• Use of OCI-based registries for sources only
  • everything as a final artefact
  • flux push artifact¹¹ is your friend

.
├── environments
│   ├── client-a
│   │   ├── prod
│   │   │   ├── services
│   │   │   │   ├ _versions.yaml
│   │   │   │   ├ foo.yaml
│   │   │   │   └ bar.yaml
│   │   │   └── system
│   │   └── stage
│   │       ├── services
│   │       └── system
│   ├── client-b
│ [...]
│
├── flux-apps
│  ├── service-stacks
│  │   ├── foo
│  │   ├── bar
│  │ [...]
│  │   └── baz
│  └── system
│    [...]
│      ├── vertical-pod-autoscaler
│      └── weaveworks-gitops

from the perspective of an individual FluxCD installation

• 0. • cloud and runtime is set up
  • provide data for stacks to become conscious
    
• 1. • load environment
  • primary Flux app
  • references all secondary service Flux apps
  • includes the version tracking ConfigMap
• 2. • load service Flux apps
  • contains relevant manifests
  • eg. OCI Sources, Terraform, Kustomization
• 3. • apply individual service apps

• FluxCD’s unique possibility to replace variables in rendered manifests before apply
• in FluxCD repo
  • environment specific _versions.yaml becomes service-versions ConfigMap
  • satisfies the “fixed versions” requirement
• In underlying IaC - basic environment information for a TF stack are written
  • base ConfigMap provides client, environment and other data
  • to form the path for Terraform state file

apiVersion: v1
kind: ConfigMap
metadata:
  name: service-versions
data:
  version_foo: "2.5.0"
  version_foo_tf: "~ 0.1.0-0"
  version_vertical_pod_autoscaler: "~> 9.0.0"
  version_vertical_pod_autoscaler_tf: "~ 0.1.0"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: init
data:
  clientId: "tenant-a"
  domain: "stage.tenant-a.tld"
  environment: "stage"
  environmentClass: "non-prod"
  region: "eu-central-1"

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: foo-iac
spec:
  interval: 5m
  provider: aws
  ref:
    semver: "${version_foo_tf}"
  url: oci://xxx.dkr.ecr.eu-central-1.amazonaws.com/iac/foo

apiVersion: infra.contrib.fluxcd.io/v1alpha2
kind: Terraform
metadata:
  name: foo
spec:
  backendConfig:
    customConfiguration: |
      backend "s3" {
        region         = "${region}"
        bucket         = "terraform-states"
        key            = "${clientId}/${environment}/stacks/foo.tfstate"
        role_arn       = "arn:aws:iam::xxx:role/tf-${clientId}-${environment}"
        dynamodb_table = "terraform-states-locks"
        encrypt        = true
      }
  sourceRef:
    kind: OCIRepository
    name: foo-iac
  vars: []

locals {
  service      = "foo"
  squad        = "bar"
  domain_name  = module.ssm_data.values["base"]["domain"]
  cluster_name = module.ssm_data.values["base"]["cluster"]["name"]
  client       = nonsensitive(module.ssm_data.values["base"]["clientId"])
  environment  = nonsensitive(module.ssm_data.values["base"]["environment"])
  env_class    = nonsensitive(module.ssm_data.values["base"]["environment_class"])

  configuration = {
    default = {
      k8s_namespace      = local.service
      k8s_sa_name        = local.service
      rds_instance_class = "db.t4g.medium"
    }
    client_a = {
      stage = {}
    }
    environment_classes = {
      non-prod  = {}
      prod = {
        rds_instance_class = "db.r6g.medium"
      }
    }
  }

  # choose the right configuration based on
  # client/environment/environment class or simply defaults
  selected_configuration = merge(
    local.configuration["default"],
    try(local.configuration[local.client][local.environment], {})
  )
}

# get the central SSM config parameters
module "ssm_data" {
  source  = "registry.example.com/foo/ssm_full_json_store/aws"
  version = "0.3.1"

  path                 = var.config_map_base_path
  include_filter_regex = "(base|foo|bar)"
}

module "database" {
  source  = "registry.example.com/foo/RDS/aws"
  version = "3.5.0"

  identifier          = local.service
  squad               = local.squad
  rds_engine_version  = local.selected_configuration["rds_engine_version"]
  rds_instance_class  = local.selected_configuration["rds_instance_class"]
  client_id           = local.client
  environment         = local.environment
  vpc_id              = module.ssm_data.values["base"]["aws"]["vpc_id"]
  subnet_ids          = module.ssm_data.values["base"]["aws"]["subnet_public_ids"]
  # [...]
}

• remember: division of concerns - cloud and runtime
• Terraform stack writes structured data as JSON
• Runtime pulls in data via External Secrets Operator¹³
• Reloader watches and upgrades Pods with their associated data

module "ssm_service_data" {
  source  = "registry.example.com/foo/ssm_json_store/aws"
  version = "1.0.2"
  
  path = "/configuration"
  name = "foo"
  data = {
    installed = true
    private = {
      database = {
        database_name     = module.database.databas
        database_username = module.database.database_username
        endpoint          = module.database.endpoint
        reader_endpoint   = module.database.reader_endpoint
        port              = module.database.cluster_port
      }
    }
    public = {}
  }
}

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: foo-secrets-ssm
spec:
  target:
    name: foo-secrets-ssm
  data:
    # [...]
    - remoteRef:
        key: /configuration/foo
        property: private.database.database_username
      secretKey: DATABASE_USER
    - remoteRef:
        key: /configuration/foo
        property: private.database.endpoint
      secretKey: DATABASE_HOST
---
kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"

• each stack has its own tf-runner pod
  • Decision: no persistent pods between runs for security reasons (permissions of SA)
• Sizing example: terraform-provider-aws_5.31.0_darwin_arm64.zip = 84MB
• NAT costs (AWS specific issue; GCP lowered egress costs to $0 recently)
  • reconcile every 30’
  • terraform init for each execution
  • 100 stacks * 48 runs/day * ~100MB providers * $0,052/GB = 480GB/$24,96 day/environment

• boring-registry to the rescue 🎉
  • caching, pull-through proxy
  • Provider Network Mirror Protocol
• provider stored and delivered as S3 objects
• 😉 use S3 VPC endpoints

• each reconcile cycle triggers one tf-runner pod per stack
• each tf-runner pods consumes
  • ~800m CPU
  • ~150M Memory
• This would spawn a lot of machines at times
• using k8s limits based on priorityClass

apiVersion: scheduling.k8s.io/v1
description: used to limit the number of terraform runners
kind: PriorityClass
metadata:
  name: terraform
value: 0 # same priority as everybody else
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: terraform-runners
spec:
  hard:
    pods: "10"
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
      - terraform

• based on Rego and similar to Open Policy Agent
• Goal: auto approve Terraform changes
  • Decision: no destroy/recreate
  • Decision: no direct IAM resources (only via controlled modules)
• ⚠️ not an easy task - talk of its own

• Backstage (B): A self-service portal to empower developers
• Argo CD (A): A GitOps-based continuous delivery (CD) tool for streamlined software delivery.
• Crossplane (C): A universal control plane simplifying self-service infrastructure provisioning through abstractions.
• Kyverno (K): A Policy as Code (PaC) tool

• existent Terraform stack and knowledge did not justify re-write of IaC
• Crossplane is bound to 1 kubernetes cluster (state in etcd) where Terraform is bound to a state file
• Introduction of Backstage was out of scope
• ArgoCD vs. FluxCD
  • ArgoCD’s handling of Helm charts (templated and applied)
  • TF-Controller as part of FluxCD eco-system
  • ArgoCD has UI and concept of multi-cluster baked in
• Kyverno “runs as a dynamic admission controller” can not be used as a decision engine