Sometimes you need to host your own Terraform Provider, and want to do it privately, without the Terraform Registry. This can be done via an S3 bucket and CloudFront, as the API just needs to respond with JSON, which can easily be done with some static JSON files. (This could most likely be done with Azure Blob Storage or GCP Cloud Storage Buckets as well).

We’ll go through:

• The Terraform to create the S3 bucket, CloudFront distributions, ACM Certificate and Route53 records.
• Build an example Terraform Provider with GoReleaser.
• Package our build to work as a registry and push it to AWS S3.
• How to consume the Terraform Provider from the Private Terraform Registry.

Code referenced in this article can be found in this GitHub repository. I started with a fork of an example provider and added what’s needed to create a registry with Terraform, and build + push the provider to it:

GitHub - cjh-cloud/terraform-provider-mock: Boilerplate template for a custom terraform provider.

Create a Git Tag

Run git tag -f v0.0.1 in the root of the repository to create a Git tag on the current commit checked out that you would like to build the provider off of. GoReleaser will use the version of this tag to create a a build.

GPG code signing

Create a GPG key to used for code-signing by Go Releaser with:
gpg --gen-key
This command will ask you for a name for the key, an eMail address to associate with the key, and a password for the key.

Export the key to a file as “armored ascii”, we will need this later when packaging the provider and pushing it to S3.

gpg --armor --export --output key.txt "{Key ID or email address}"

Set an environment variable for the GPG Fingerprint in CLI, which will be part of the output when you create the key. This will be used by Go Releaser.

export GPG_FINGERPRINT="[~40 character fingerprint]"

.goreleaser.yaml

Create a .goreleaser.yaml file in the root of your repository with the following content. This file uses the GPG_FINGERPRINT environment variable which is how GoReleaser knows which key to sign it with.

---
builds:
- env:
  - CGO_ENABLED=0
  mod_timestamp: "{{ .CommitTimestamp }}"
  flags:
    - -trimpath
  ldflags:
    - "-s -w -X main.version={{.Version}} -X main.commit={{.Commit}}"
  goos:
    - freebsd
    - windows
    - linux
    - darwin
  goarch:
    - amd64
    - "386"
    - arm
    - arm64
  ignore:
    - goos: darwin
      goarch: "386"
  binary: "{{ .ProjectName }}_v{{ .Version }}"
archives:
  - format: zip
    name_template: "{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}"
checksum:
  name_template: "{{ .ProjectName }}_{{ .Version }}_SHA256SUMS"
  algorithm: sha256
signs:
  - artifacts: checksum
    args:
      - "--batch"
      - "--local-user"
      - "{{ .Env.GPG_FINGERPRINT }}"
      - "--output"
      - "${signature}"
      - "--detach-sign"
      - "${artifact}"

Build locally with goreleaser release --skip-publish , which will create a dist directory. Go Releaser will ask you for the GPG key password while building, so that it can do the code-signing step properly.
Note: You don’t have to skip the publish step, set a GitHub token in the CLI for Go Releaser to also push the build files to GitHub as part of a release.

Creating AWS Resources

We’ll create the following AWS resources to host the Terraform provider:

• S3 bucket to store the provider build files.
• CloudFront distribution to add a domain name.
• ACM Certificate for SSL on the CF distribution.
• Route53 records (assumed that host zone already exists).

Change the local values to fit your needs, and run terraform init and terraform apply to create the resources. Make sure you have AWS credentials set either in a ~/.aws directory, or the current terminal session via environment variables. The bucket is set to public so that all files can be reached through the CloudFront distribution.

provider "aws" {
  region = "us-east-1"
}

locals {
  name           = "TF Provider"
  domain_name    = "cjscloud.city"
  subdomain_name = "tfp.${local.domain_name}"
  environment    = "default"
}

# S3 bucket
resource "aws_s3_bucket" "provider" {
  bucket = local.subdomain_name

  tags = {
    Name        = local.name
    Environment = local.environment
  }
}

resource "aws_s3_bucket_public_access_block" "provider" {
  bucket = aws_s3_bucket.provider.id

  block_public_acls       = false
  block_public_policy     = false
  ignore_public_acls      = false
  restrict_public_buckets = false
}

resource "aws_s3_bucket_policy" "provider" {
  bucket = aws_s3_bucket.provider.id
  policy = data.aws_iam_policy_document.provider.json
}

data "aws_iam_policy_document" "provider" {
  statement {
    principals {
      type        = "*"
      identifiers = ["*"]
    }

    actions = [
      "s3:GetObject",
    ]

    resources = [
      "${aws_s3_bucket.provider.arn}/*",
    ]
  }
}

# ACM Certificate
resource "aws_acm_certificate" "provider" {
  domain_name       = local.subdomain_name
  validation_method = "DNS"

  tags = {
    Name        = local.name
    Environment = local.environment
  }

  lifecycle {
    create_before_destroy = true
  }
}

# Route53 Records
data "aws_route53_zone" "provider" {
  name         = local.domain_name
  private_zone = false
}

resource "aws_route53_record" "provider" {
  for_each = {
    for dvo in aws_acm_certificate.provider.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = data.aws_route53_zone.provider.zone_id
}

resource "aws_route53_record" "www" {
  zone_id = data.aws_route53_zone.provider.zone_id
  name    = local.subdomain_name
  type    = "CNAME"
  ttl     = 300
  records = [aws_cloudfront_distribution.provider.domain_name]
}

# Cert DNS validation, depended on by CloudFront
resource "aws_acm_certificate_validation" "provider" {
  certificate_arn         = aws_acm_certificate.provider.arn
  validation_record_fqdns = [for record in aws_route53_record.provider : record.fqdn]
}

# Cloudfront
resource "aws_cloudfront_distribution" "provider" {
  origin {
    domain_name = aws_s3_bucket.provider.bucket_regional_domain_name
    origin_id   = aws_s3_bucket.provider.bucket_domain_name
  }

  enabled             = true
  is_ipv6_enabled     = true
  comment             = "Terraform Provider Registry"
  default_root_object = "index.html"

  aliases = [local.subdomain_name]

  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD", "OPTIONS"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = aws_s3_bucket.provider.bucket_domain_name

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "allow-all"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
  }

  price_class = "PriceClass_100"

  restrictions {
    geo_restriction {
      # restriction_type = "whitelist"
      # locations        = ["NZ"]
      restriction_type = "none"
    }
  }

  tags = {
    Name        = local.name
    Environment = local.environment
  }

  viewer_certificate {
    cloudfront_default_certificate = false
    acm_certificate_arn            = aws_acm_certificate.provider.arn
    ssl_support_method             = "sni-only"
  }
}

How to package the Terraform provider

The following Go code can package the build in the dist/ directory and create the necessary files for a Terraform provider registry hosted as static files. In the example repo, this script is under a folder named tfpp/ and by default assumes there is a dist/ folder and key.txt file in the parent directory, but different paths can be provided via command line flags.

An example of calling this would be like:

go run main.go \
  -ns=cjh-cloud \
  -d=tfp.cjscloud.city \
  -gf=$GPG_FINGERPRINT \
  -v=0.0.1

main.go

Here’s the Go script that will package a Terraform provider build to act as a Terraform Registry, hosted statically. This is in the GitHub repo mentioned at the start as in the tfpp/ (Terraform Provider Packager) directory.

package main

import (
 "bufio"
 "flag"
 "fmt"
 "io"
 "log"
 "os"
 "strings"
)

func main() {
 log.Println("📦 Packaging Terraform Provider for private registry...")

 namespace := flag.String("ns", "", "Namespace for the Terraform registry.") // No default
 domain := flag.String("d", "", "Private Terraform registry domain.") // No default
 providerName := flag.String("p", "mock", "Name of the Terraform provider.")
 distPath := flag.String("dp", "../dist", "Path to Go Releaser build files.")
 repoName := flag.String("r", "terraform-provider-mock", "Name of the provider repository used in Go Releaser build name.")
 version := flag.String("v", "0.0.1", "Semantic version of build.")
 gpgFingerprint := flag.String("gf", "", "GPG Fingerprint of key used by Go Releaser") // No default
 gpgPubKeyFile := flag.String("gk", "../key.txt", "Path to GPG Public Key in ASCII Armor format.")
 flag.Parse()

 *distPath = *distPath + "/" // If the path already ends in "/", it shouldn't matter

 // Create release dir - only the contents of this need to be uploaded to S3
 err := createDir("release")
 if err != nil {
  log.Printf("Error creating 'release' dir: %s", err)
 }

 // Create .wellKnown directory and terraform.json file
 err = wellKnown()
 if err != nil {
  log.Printf("Error creating '.wellKnown' dir: %s", err)
 }

 // Create v1 directory
 err = provider(*namespace, *providerName, *distPath, *repoName, *version, *gpgFingerprint, *gpgPubKeyFile, *domain)
 if err != nil {
  log.Printf("Error creating 'v1' dir: %s", err)
 }

 log.Println("📦 Packaged Terraform Provider for private registry.")
}

// This establishes the "API" as a TF provider by responding with the correct JSON payload, by using static files
func wellKnown() (error) {
 log.Println("* Creating .well-known directory")

 err := createDir("release/.well-known")
 if err != nil {
  return err
 }

 terraformJson := []byte(`{"providers.v1": "/v1/providers/"}`)

 log.Println("  - Writing to .well-known/terraform.json file")
 err = writeFile("release/.well-known/terraform.json", terraformJson)
 if err != nil {
  return err
 }

 return nil
}

// provider is the Terraform name
// repoName is the Repository name
func provider(namespace, provider, distPath, repoName, version, gpgFingerprint, gpgPubKeyFile, domain string) (error) {
 // Path to semantic version dir
 versionPath := providerDirs(namespace, provider, version)

 // Files to create under v1/providers/[namespace]/[provider_name]
 createVersionsFile(namespace, provider, distPath, repoName, version) // Creates version file one above download, which is why downloadPath isn't used

 // Files/Directories to create under v1/providers/[namespace]/[provider_name]/[version]
 copyShaFiles(versionPath, distPath, repoName, version)
 downloadPath, err :=  createDownloadsDir(versionPath)
 if err != nil {
  return err
 }

 // Create darwin, freebsd, linux, windows dirs
 createTargetDirs(*downloadPath)

 // Copy all zips
 copyBuildZips(*downloadPath, distPath, repoName, version)

 // Create all individual files for build targets and each architecture for the build targets
 createArchitectureFiles(namespace, provider, distPath, repoName, version, gpgFingerprint, gpgPubKeyFile, domain)

 return nil
}

// Create the directories with a path format v1/providers/[namespace]/[provider_name]/[version]
func providerDirs(namespace, repoName, version string) string {
 log.Println("* Creating release/v1/providers/[namespace]/[repo]/[version] directories")

 providerPathArr := [6]string{"release", "v1", "providers", namespace, repoName, version}

 var currentPath string
 for _, v := range providerPathArr {
  currentPath = currentPath + v + "/"
  createDir(currentPath)
 }

 return currentPath
}

// Create the versions file under v1/providers/[namespace]/[provider_name]
func createVersionsFile(namespace, provider, distPath, repoName, version string) (error) {
 log.Println("* Writing to release/v1/providers/[namespace]/[repo]/versions file")

 versionPath := fmt.Sprintf("release/v1/providers/%s/%s/versions", namespace, provider)

 shaSumContents, err := getShaSumContents(distPath, repoName, version)
 if err != nil {
  return err
 }

 // Build the versions file...
 platforms := ""
 for _, line := range shaSumContents {
  fileName := line[1] // zip file name

  // get os and arch from filename
  removeFileExtension := strings.Split(fileName, ".zip")
  fileNameSplit := strings.Split(removeFileExtension[0], "_")

  // Get build target and architecture from the zip file name
  target := fileNameSplit[2]
  arch := fileNameSplit[3]

  platforms += "{"
    platforms += fmt.Sprintf(`"os": "%s",`, target)
    platforms += fmt.Sprintf(`"arch": "%s"`, arch)
  platforms += "}"
  platforms += ","
 }
 platforms = strings.TrimRight(platforms, ",") // remove trailing comma, json does not allow

 var versions = []byte(fmt.Sprintf(`
{
  "versions": [
    {
      "version": "%s",
      "protocols": [
        "4.0",
        "5.1"
      ],
      "platform": [
        %s
      ]
    }
  ]
}
`, version, platforms))

 writeFile(versionPath, versions)

 return nil
}

func copyShaFiles(destPath, srcPath, repoName, version string) {
 log.Printf("* Copying SHA files in %s directory", srcPath)

 // Copy files from srcPath 
 shaSum := repoName + "_" + version + "_SHA256SUMS"
 shaSumPath := srcPath + "/" + shaSum

 // _SHA256SUMS file
 _, err := copyFile(shaSumPath, destPath + shaSum)
 if err != nil {
  log.Println(err)
 }

 // _SHA256SUMS.sig file
 _, err = copyFile(shaSumPath + ".sig", destPath + shaSum + ".sig")
 if err != nil {
  log.Println(err)
 }
}

func createDownloadsDir(destPath string) (*string, error) {
 log.Printf("* Creating download/ in %s directory", destPath)

 downloadPath := destPath + "download/"
 
 err := createDir(downloadPath)
 if err != nil {
  return nil, err
 }

 return &downloadPath, nil
}

func createTargetDirs(destPath string) (error) {
 log.Printf("* Creating target dirs in %s directory", destPath)

 targets := [4]string{"darwin", "freebsd", "linux", "windows"}

 for _, v := range targets {
  err := createDir(destPath + v)
  if err != nil {
   return err
  }
 }

 return nil
}

func copyBuildZips(destPath, distPath, repoName, version string) (error) {
 log.Println("* Copying build zips")

 shaSumContents, err := getShaSumContents(distPath, repoName, version)
 if err != nil {
  return err
 }

 // Loop through and copy each
 for _, v := range shaSumContents {
  zipName := v[1]
  zipSrcPath := distPath + zipName
  zipDestPath := destPath + zipName

  log.Printf("  - Zip Source: %s", zipSrcPath)
  log.Printf("   - Zip Dest:  %s", zipDestPath)
  
  // Copy the zip
  _, err := copyFile(zipSrcPath, zipDestPath)
  if err != nil {
   return err
  }
 }

 return nil
}

func getShaSumContents(distPath, repoName, version string) ([][]string, error) {
 shaSumFileName := repoName + "_" + version + "_SHA256SUMS"
 shaSumPath := distPath + "/" + shaSumFileName

 shaSumLine, err := readFile(shaSumPath)
 if err != nil {
  return nil, err
 }

 buildsAndShaSums := [][]string{}

 for _, line := range shaSumLine {
  lineSplit := strings.Split(line, "  ")

  row := []string{lineSplit[0], lineSplit[1]}
  buildsAndShaSums = append(buildsAndShaSums, row)
 }

 // log.Println(buildsAndShaSums)

 return buildsAndShaSums, nil
}

// Create architecture files for each build target
func createArchitectureFiles(namespace, provider, distPath, repoName, version, gpgFingerprint, gpgPubKeyFile, domain string) (error) {
 log.Println("* Creating architecure files in target directories")

 // filename = terraform-provider-[provider]_0.0.1_darwin_amd64.zip - provider_name + version + target + architecture + .zip
 prefix := fmt.Sprintf("v1/providers/%s/%s/%s/", namespace, provider, version)
 pathPrefix := fmt.Sprintf("release/%s", prefix)
 urlPrefix := fmt.Sprintf("https://%s/%s", domain, prefix)

 // download url = https://example.com/v1/providers/namespace/provider/0.0.1/download/terraform-provider_0.0.1_darwin_amd64.zip
 downloadUrlPrefix := urlPrefix + "download/"
 downloadPathPrefix := pathPrefix + "download/"

 // shasums url = https://example.com/v1/providers/namespace/provider/0.0.1/terraform-provider_0.0.1_SHA256SUMS
 shasumsUrl := urlPrefix + fmt.Sprintf("%s_%s_SHA256SUMS", repoName, version)
 // shasums_signature_url = https://example.com/v1/providers/namespace/provider/0.0.1/terraform-provider_0.0.1_SHA256SUMS.sig
 shasumsSigUrl := shasumsUrl + ".sig"

 shaSumContents, err := getShaSumContents(distPath, repoName, version)
 if err != nil {
  return err
 }

 // Get contents of GPG key
 gpgFile, err := readFile(gpgPubKeyFile)
 if err != nil {
  log.Printf("Error reading '%s' file: %s", gpgPubKeyFile, err)
 }

 // loop through every line and stick with \\n
 gpgAsciiPub := ""
 for _, line := range gpgFile {
  gpgAsciiPub = gpgAsciiPub + line + "\\n"
 }
 // log.Println(gpgAsciiPub)

 for _, line := range shaSumContents {
  shasum := line[0] // shasum of the zip
  fileName := line[1] // zip file name

  downloadUrl := downloadUrlPrefix + fileName

  // get os and arch from filename
  removeFileExtension := strings.Split(fileName, ".zip")
  fileNameSplit := strings.Split(removeFileExtension[0], "_")

  // Get build target and architecture from the zip file name
  target := fileNameSplit[2]
  arch := fileNameSplit[3]
  
  // build filepath
  archFileName := downloadPathPrefix + target + "/" + arch 

  var architectureTemplate = []byte(fmt.Sprintf(`
{
  "protocols": [
    "4.0",
    "5.1"
  ],
  "os": "%s",
  "arch": "%s",
  "filename": "%s",
  "download_url": "%s",
  "shasums_url": "%s",
  "shasums_signature_url": "%s",
  "shasum": "%s",
  "signing_keys": {
    "gpg_public_keys": [
      {
        "key_id": "%s",
        "ascii_armor": "%s",
        "trust_signature": "",
        "source": "",
        "source_url": ""
      }
    ]
  }
}
`, target, arch, fileName, downloadUrl, shasumsUrl, shasumsSigUrl, shasum, gpgFingerprint, gpgAsciiPub))

  log.Printf("  - Arch file: %s", archFileName)

  err := writeFile(archFileName, architectureTemplate)
  if err != nil {
   return err
  }
 }

 return nil
}

func createDir(path string) (error) {
 err := os.Mkdir(path, os.ModePerm)
 return err
}

func copyFile(src, dst string) (int64, error) {
 sourceFileStat, err := os.Stat(src)
 if err != nil {
  return 0, err
 }

 if !sourceFileStat.Mode().IsRegular() {
  return 0, fmt.Errorf("%s is not a regular file", src)
 }

 source, err := os.Open(src)
 if err != nil {
  return 0, err
 }
 defer source.Close()

 destination, err := os.Create(dst)
 if err != nil {
  return 0, err
 }
 defer destination.Close()
 nBytes, err := io.Copy(destination, source)
 return nBytes, err
}

func readFile(filePath string) ([]string, error) {
 readFile, err := os.Open(filePath)

 if err != nil {
  return nil, err
 }
 fileScanner := bufio.NewScanner(readFile)
 fileScanner.Split(bufio.ScanLines)
 var fileLines []string

 for fileScanner.Scan() {
  fileLines = append(fileLines, fileScanner.Text())
 }

 readFile.Close()

 return fileLines, nil
}

func writeFile(fileName string, fileContents []byte) (error) {
 err := os.WriteFile(fileName, fileContents, 0644)
 return err
}

This will create a release/ directory, which can be pushed to the S3 bucket with something like aws s3 sync release s3://tfp.cjscloud.city. The directory structure of the release will look something like in the following image.

Using the provider

Define the provider and set the source and version under required_providers, as well as defining the provider object.

terraform {
  required_providers {
    mock = {
      source  = "tfp.cjscloud.city/cjh-cloud/mock"
      version = "0.0.1"
    }
  }
}

provider "mock" {
  foo = "test_foo_value"
}

resource "mock_example" "testing" {
  not_computed_required = "some value"

  dynamic "foo" {
    for_each = [{ number = 1 }, { number = 2 }, { number = 3 }]
    content {
      bar {
        number = foo.value.number
      }
    }
  }
  /*
   * The above is equivalent to:
   *
   * foo {
   *   bar {
   *     number = 1
   *   }
   * }
   * foo {
   *   bar {
   *     number = 2
   *   }
   * }
   * foo {
   *   bar {
   *     number = 3
   *   }
   * }
  */

  dynamic "baz" {
    // The variable inside the for_each block doesn't have to be the same as 
    // what you're assigning the value to.
    for_each = [{ something = "x" }, { something = "y" }, { something = "z" }]
    content {
      qux = baz.value.something
    }
  }
  /*
   * The above is equivalent to:
   *
   * baz {
   *   qux = "x"
   * }
   * baz {
   *   qux = "y"
   * }
   * baz {
   *   qux = "z"
   * }
  */

  some_list = ["a", "b", "c"]
}

output "last_updated" {
  value = mock_example.testing.last_updated
}

Run terraform init to download the provider from the private registry, which should successfully initialise. Then run terraform apply to create a resource using the provider.

Cleanup

Empty the S3 bucket of files, then run terraform destroy to remove all the AWS resources when tearing down the private registry.

If you found this useful, please consider following!
Cheers,

Terraform workspaces simplify having multiple environments with the same backend. They also improve developer experience by making it easier to create new workspaces, and switch between them using CLI commands.

Code for this article can be found here:

GitHub - cjh-cloud/lambda-chain-tf: Example of a multiple AWS Lambdas, with Terraform Workspace S3 backend

Set up S3 backend resources

An S3 bucket will be created to store the Terraform state files for every workspace. We will also enable object versioning on the bucket, so that we can retrieve past versions of the Terraform state if we need to role back. Server side encryption is turned on for security.

A DynamoDB table is also created to allow the Terraform backend to handle state locking, to protect the state from concurrent plans or applies, allowing a team to potentially work together on this, as well as CI integration. Deletion protection is also enabled so that the table does not get accidentally deleted. Remember to set this to false and do an apply when cleaning everything up afterwards, otherwise Terraform won’t be able to delete the table.

The count to only create if the current selected workspace is default, is not necessary, only if you’re creating other workspaces in the same directory (which we won’t be doing this time around), as we only need one backend.

backend_res.tf

resource "aws_s3_bucket" "backend" {
  count = terraform.workspace == "default" ? 1 : 0

  bucket_prefix = "terraform-backend"

  tags = {
    Name        = "Terraform Backend"
    Environment = terraform.workspace
  }
}

output "backend_bucket_name" {
  value = aws_s3_bucket.backend.0.id
}

resource "aws_s3_bucket_versioning" "backend" {
  count = terraform.workspace == "default" ? 1 : 0

  bucket = aws_s3_bucket.backend.0.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "backend" {
  count = terraform.workspace == "default" ? 1 : 0

  bucket = aws_s3_bucket.backend.0.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_dynamodb_table" "terraform_lock" {
  count = terraform.workspace == "default" ? 1 : 0

  name                        = "terraform_state"
  deletion_protection_enabled = true
  read_capacity               = 5
  write_capacity              = 5
  hash_key                    = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
  tags = {
    Name        = "Terraform Backend"
    Environment = terraform.workspace
  }
}

Run a terraform apply to create the resources. After the resources are created, there should be an output section with the name of the S3 bucket created, this will be used in the backend.

Migrate the state to the S3 remote backend

replace the bucket value with the output from above.
Change region to suit your needs, unfortunately variables are not accepted here, so we can’t use var.region .

backend.tf

terraform {

  # Comment this out when initialising resources.
  backend "s3" {
    region         = "ap-southeast-2"
    bucket         = "terraform-backend20231013052848766400000001"
    key            = "services/tf-workspace-ex/default.tfstate"
    dynamodb_table = "terraform_state"
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.11.0"
    }
  }
}

provider "aws" {
  region = var.region
}

variable "region" {
  description = "AWS Region"
  default     = "ap-southeast-2"
  type        = string
}

Once the above backend.tf file is created, run terraform init . Since the current state for the S3 and DynamoDB resources is in a local terraform.tfstate file, it will ask if you would like to copy state to the remote backend, select yes.

✅ The remote backend is finished, we can now start creating environments and Terraform code for our project resources.

Create a project that uses the backend and Terraform Workspaces

Workspaces are a great way to define infrastructure once, and create multiple environments off of it without needing to change the configuration in the backend block.

Make a project directory in the same directory as the existing Terraform backend resources.

Note: You don’t need to create a project directory, you could do everything within the current directory. If you go down this route, I recommend not creating resources within the default workspace, and using that only for the backend state resources.

Move into the project dir with cd project and add some new Terraform resources here for your project.

Create another backend.tf file too, the only thing slightly different to the previous one above is the key, to make sure the state file in the bucket has a different name:

terraform {

  backend "s3" {
    region         = "ap-southeast-2"
    bucket         = "terraform-backend20231013052848766400000001"
    key            = "services/tf-workspace-ex/project.tfstate"
    dynamodb_table = "terraform_state"
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.11.0"
    }
  }
}

provider "aws" {
  region = var.region
}

Run terraform init and then create a new workspace for a development environment with terraform workspace new dev .

Apply the terraform with terraform apply -var-file=./vars/dev.tfvars . Note: I’m using var files for each environment, so leave that part off of the command if you aren’t.

Create a new workspace for a test environment terraform workspace new test .

Apply the terraform for the test environment terraform apply -var-file=./vars/test.tfvars .

View workspaces with terraform workspace list .

Select dev workspace again with terraform workspace select dev .

You’ve now created resources for a Dev and Test environment, with the Terraform state stored remotely in an S3 bucket.

Clean up

Run terraform destroy -var-file=./vars/dev.tfvars and switch to Test with terraform workspace select test and run terraform destroy -var-file=./vars/test.tfvars. This cleans up all the resources in the project directory.

Move up one directory to where the backend resources are and disable deletion protection on the DynamoDB table if need be. (set this to false in the backend_res.tf file and do an apply ). Manually delete the contents in the S3 bucket from the AWS console, as Terraform cannot destroy non-empty buckets.

Run terraform destroy . Everything should now be cleaned up 👍.

I hope you found that useful!
Thanks for reading, cheers 🤙

Automate versioning of Terraform in a GitHub repo with the use of GitHub Actions and Semantic Release. GitHub Actions will automate the process of running Semantic Release on commits to a chosen branch, and Semantic release will create a Git Tag, and a GitHub Release with release notes about changes in the added commits.

This action will checkout your repo, setup node, install semantic release (this could be cached using NPM cache, to speed up future runs), run sematic-release with a GitHub token, so that the action has permissions to create a release and tag for the repo.

Code included in this post can be found here:

GitHub - cjh-cloud/lambda-chain-tf: Example of a multiple AWS Lambdas, with Terraform Workspace S3 backend

.github/workflows/release.yaml

This file defines the GitHub Action:

• Runs on push to prod branch, change this to fit.
• Sets permissions to be able to create releases
• Checkout repo, setup node, install Semantic Release and related plugins, run Semantic Release with GitHub token.
• GITHUB_TOKEN — you don’t need to set up the secret, that’s automagical, and handles by GitHub Actions. The token is needed so that Semantic Release is able to create and push a Git Tag, as well as create the GitHub Release.

name: GitHub Release

on:
  push:
    branches:
      - prod

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  release:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    permissions:
      contents: write
      issues: read

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3

      - name: Setup Node.js environment
        uses: actions/setup-node@v3.7.0
        with:
          # Version Spec of the version to use. Examples: 12.x, 10.15.1, >=10.15.0.
          node-version: 18.x # optional

      - name: Install Semantic Release and plugins
        run: |
          npm install semantic-release @semantic-release/exec @semantic-release/changelog @semantic-release/github

      - name: Create Release
        run: npx semantic-release
        env: 
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Next it our config for semantic release, so that it can work out how to bump the version, based on the commits being merged into the branch.

.releaserc.yaml

Important parts are:

• branches — Here is is set to prod , but change this to suit your needs.
• analyzeCommits — decides how to bump semantic version based on commit messages in the merging branch. E.g. commits beginning with feat: are considered a minor release, and fix: is considered a patch.
• The exec command — outputs the new version to a file

---
branches:
  - prod
defaultBranch: "prod"
preset: "angular"
tagFormat: "${version}"
plugins:
  - "@semantic-release/commit-analyzer"
  - "@semantic-release/release-notes-generator"
  - "@semantic-release/changelog"
  - "@semantic-release/github"
  - "@semantic-release/exec"

verifyConditions:
  - "@semantic-release/github"

analyzeCommits:
  - path: "@semantic-release/commit-analyzer"
    releaseRules:
    - type: "feat"
      release: "minor"
    - type: "fix"
      release: "patch"
    - type: "breaking"
      release: "major"
    - type: "refactor"
      release: "patch"
    - type: "chore"
      release: "patch"

generateNotes:
  - path: "@semantic-release/release-notes-generator"
    writerOpts:
      groupBy: "type"
      commitGroupsSort:
        - "breaking"
        - "feat"
        - "fix"
        - "chore"
      commitsSort: "header"
    types:
    - type: "feat"
    - section: "Features"
    - type: "fix"
    - section: "Bug Fixes"
    - type: "breaking"
    - section: "Breaking Changes"
    - type: "refactor"
    - hidden: true
    - type: "chore"
    - section: "Chore"
    - type: "docs"
    - hidden: true
    - type: "doc"
    - hidden: true
    - type: "style"
    - hidden: true
    - type: "perf"
    - hidden: true
    - type: "test"
    - hidden: true
    presetConfig: true
  - path: "@semantic-release/exec"
    cmd: "echo ${nextRelease.version} > nextRelease"
prepare:
  - path: "@semantic-release/changelog"
    changelogFile: "docs/CHANGELOG.md"
publish:
  - path: "@semantic-release/github"

success:
  - "@semantic-release/github"

fail:
  - "@semantic-release/github"

Once these files have been committed and pushed to the branch specified in the above files ( prod in this case), either directly to the branch, or merge, GitHub Actions will trigger a run and create a release.

Releases for a GitHub repo can be viewed at /releases path on the URL for the repository.

I hope you found this useful!
Cheers 🤙,

The following link shows how to deploy the ADOT Collector to EKS Fargate that enables the collection of Container Insights metrics from EKS Fargate workloads and send them to CloudWatch.

This is deployed via a couple YAML files that require some editing to work for your cluster. This article will go through deploying the same ADOT collector via a Helm chart, as well some of the prerequisites required for ADOT to run.

Container Insights EKS Fargate | AWS Distro for OpenTelemetry

Prerequisites

It is assumed you have an EKS Fargate Cluster running.

The following resources are created with Terraform as an example, but they could be done through CLI or another method.

Cert Manager

You will need to install Cert Manager for ADOT to work. More info on why this is required.

Change the webhook port for it to work on Fargate, as the Kubelet listens on port 10250 by default and clashes with Cert Manager.

resource "helm_release" "cert_manager" {
  depends_on = [
    aws_eks_addon.coredns
  ]

  name             = "cert-manager"
  chart            = "cert-manager"
  create_namespace = true
  repository       = "https://charts.jetstack.io"
  version          = "1.11.0"
  namespace        = "cert-manager"

  set {
    name  = "startupapicheck.timeout"
    value = "5m"
  }
  set {
    name  = "installCRDs"
    value = "true"
  }
  set {
    name  = "webhook.securePort"
    value = "10260"
  }
}

AWS IAM Role for ADOT Service Account

You’ll also need to set up an AWS IAM role for the ADOT Service Account to use with the following:

• a trust policy that provides federated access to the AWS EKS cluster through OIDC (I have used the “terraform-aws-modules/eks/aws” Terraform module and am able to get the OIDC provider arn via module.eks.oidc_provider_arn)
• The managed policy CloudWatchAgentServerPolicy so that ADOT has permissions to send Container Insights to CloudWatch.

variable "service_account_namespace" {
  default = "fargate-container-insights"
}

variable "service_account_name" {
  default = "adot-collector"
}

data "aws_iam_policy_document" "adot_trust_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [module.eks.oidc_provider_arn]
    }
    condition {
      variable = "${module.eks.oidc_provider}:sub"
      test     = "StringEquals"
      values   = ["system:serviceaccount:${var.service_account_namespace}:${var.service_account_name}"]
    }
  }
}

resource "aws_iam_role" "adot_service_acc" {

  name = "${var.env_name}-adot-role"

  assume_role_policy = data.aws_iam_policy_document.adot_trust_policy.json

  managed_policy_arns = [
    "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy",
  ]
}

output "adot_iam_role_arn" {
  value = aws_iam_role.adot_service_acc.arn
}

resource "kubernetes_namespace" "fargate_container_insights" {
  metadata {
    name = var.service_account_namespace
  }
}

The output adot_iam_role_arn will output the ARN of the IAM role which will be used in the ADOT Helm chart.

Installation of the ADOT Fargate Helm chart

Add the AWS Distro for OpenTelemetry Helm repository to Helm:

helm repo add adot-fargate https://github.com/cjh-cloud/adot-fargate  
helm repo update

Install the AWS Distro for OpenTelemetry Helm chart:
Change the cluster name, region to your own EKS cluster and IAM role ARN to the value of the adot_iam_role_arn Terraform output above.

helm install adot-fargate cjh-cloud/adot-fargate \
  --set clusterName=my-cluster \
  --set region=ap-southeast-2 \
  --set serviceAccount.iamRoleArn=arn:aws:iam::[ACCOUNT ID]:role/adot-role

OR Install via repo:

helm upgrade --install aws-distro-open-telemetry ./ \
  --set clusterName=my-cluster \
  --set region=ap-southeast-2 \
  --set serviceAccount.iamRoleArn=arn:aws:iam::[ACCOUNT ID]:role/adot-role

Verify that the AWS Distro for OpenTelemetry deployment is running:

kubectl get pods -n fargate-container-insights

The output should show a pod with the name adot-collector-0.
View "Container Insights" under AWS CloudWatch to view EKS Cluster metrics.

Thanks for reading, I hope you found this useful!

In this post, I’ll go over how to use a GitHub Action to build and push a runner image to GitHub Packages, and then use said runner image as the base for another action. These two actions will be part of the same GitHub repository.

All code referenced is also in this repository can be found here:

GitHub - cjh-cloud/collective at terraform

In this example, I have some Terraform in the branch that I want to be able to run plans on within a GitHub action. The GitHub ubuntu-latest runner does not have Terraform installed, which means we could either install it in the pipeline. However, it could be more efficient to use an image that has Terraform, and any other tools we need, pre-installed.

We could define this image with a Dockerfile in our repo, and have an action that builds this image and pushes it to GitHub packages only when there are changes to that Dockerfile. (As we don’t need to build a new image every time there are changes to the whole repository).

tooling.yaml — Workflow to create the tooling image

This is a pretty simple action that will only run on commits that have changed to the Dockerfile in the tooling directory. It will build a Docker image and push it to GitHub packages in the same repo. The image will have the name of the repo, and be tagged with the branch name. The name of this repo is collective and all this code is in the terraform branch, so the image name with be ghcr.io/cjh-cloud/collective:terraform.

We don’t need to set up a secret for GITHUB_TOKEN (line 28) in the repository as GitHub automates the authorisation for us during the workflow run.

We can see within GitHub that we now have a package 📦 as part of the repo:

infra.yaml — Workflow to use the tooling image to run Terraform

The terraform job uses the container parameter to set the Docker image to run the job on, in this case the image we created with the previous Action. We can then checkout the repo and run Terraform commands in the pipeline, since the image that was built has Terraform installed.

It’s possible to create any kind of image with the tools you need installed, thereby saving time for the workflow that uses this image by eliminating the need to install these tools each time it runs.

If this helped you in any way, please consider giving me a follow. I’m trying to get to 100 followers, so it would mean a lot!

Cheers 🤙

I found that I couldn’t configure Jenkins to use a proxy server for Docker. However, I could spin up a Docker container within the pipeline that could. This is how to do that…

Have a docker in docker image with the AWS cli installed that can be puled by Jenkins. In the example below I’ve used “docker_tooling:latest” to represent this.

This Jenkinsfile consists of a stage that will spin up a “dind” container with a configured proxy, which can then be used in later steps by running docker exec commands against it.

This isn’t a fantastic solution, but helps if you’re in a pickle.

I hope you found this useful!
Cheers 🤙

I’ve found that when trying to run the docker-in-docker image with rootless, I get the following error:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Turns out this is because the path to the Docker Host is actually unix:///run/user/1000/docker.sock and this was unfortunately left out of the documentation in the Docker docs (at the time of writing).

To get around this, you can use the following Dockerfile

FROM docker:20.10.10-dind-rootless

ENV DOCKER_HOST=unix:///run/user/1000/docker.sock

and then this docker-compose.yml file:

version: '3.8'

services:
  dindtest:
    build:
      context: .
    privileged: true

and then run docker-compose up and open another terminal and run docker exec -it [CONTAINER NAME] sh and you should be able to start running docker commands from within the container.

I hope you found this useful.
Chur 🤙,