Solving Database Management Chaos with Kubernetes Custom Resource Definitions: A Practical Guide

The Problem: Database Provisioning Nightmare

Picture this: It’s Friday morning, and your development team is scrambling to deploy three new microservices. Each service needs its own database instance with specific configurations — different storage sizes, backup schedules, and performance tiers. Your current process involves:

• Manually creating database instances through cloud provider consoles (or fine maybe scripts)
• Writing custom deployment scripts for each database type
• Managing inconsistent naming conventions across environments
• Coordinating with the infrastructure team for every single database request
• Tracking database lifecycles through spreadsheets (yes, spreadsheets!)

Sound familiar? This chaos is exactly what many engineering teams face when managing databases in Kubernetes environments.

Another familiar example is that we have multiple running deployments with a large number of pods, services etc and we need to do an upgrade, or maybe different pods seem to be going down at different times. Now, managing resources with multiple moving parts becomes easier with abstraction — exactly how deployment is easier to manage than individual pods.

The solution? Custom Resource Definitions (CRDs) — Kubernetes’ powerful mechanism for extending the API with your own resource types.

What Are Custom Resource Definitions?

Custom Resource Definitions allow you to define your own Kubernetes resources beyond the standard pods, services, and deployments. Think of CRDs as blueprints that teach Kubernetes about new types of objects your applications need — in our case, databases.

When you create a CRD, you’re essentially extending Kubernetes’ vocabulary. Instead of manually provisioning databases, you can declare them as Kubernetes resources, leveraging the same declarative approach you use for everything else in your cluster. Think of it like how Kubernetes understands what deployment is, we can make it start to understand what database is too.

Key Concepts You Need to Know

Custom Resource (CR): An instance of your custom resource definition. If CRD is the blueprint, CR is the actual house built from that blueprint.

API Group: A collection of related resources. For example, database.company.com groups all our database-related resources.

Schema Validation: Rules that ensure your custom resources have valid configurations before they’re accepted by Kubernetes. Much like when you do “kubectl edit ..” and it denies editing the file when you have an incorrect tab space.

Controller Pattern: The mechanism that watches for changes to your custom resources and takes action to maintain the desired state (spec in yaml).

The Solution Architecture: Database-as-a-Resource

We’ll build a database management system using CRDs that allows developers to request databases like this:

apiVersion: database.company.com/v1
kind: Database
metadata:
  name: user-service-db
  namespace: production
spec:
  type: postgresql
  version: "13"
  storage: 100Gi

Let’s implement this step by step.

1. We’ll create CRD, and then an instance of that CRD (Database)
2. We’ll understand what controllers and operators are
3. We go on to see how use these with and without controllers
4. And lastly, talk about some advanced CRD features and common pitfalls

Step 1: Defining Our Database CRD

First, we’ll create the CRD that defines what a “Database” resource looks like:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.database.company.com
spec:
  group: database.company.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                type:
                  type: string
                  enum: ["postgresql", "mysql", "mongodb"]
                  description: "Database engine type"
                version:
                  type: string
                  pattern: '^[0-9]+(\.[0-9]+)*$'
                  description: "Database version"
                storage:
                  type: string
                  pattern: '^[0-9]+[KMGT]i$'
                  description: "Storage size (e.g., 10Gi, 500Mi)"
              required:
                - type
                - version
                - storage
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames:
      - db

Let’s break down the important parts:

API Group and Version: database.company.com/v1 creates a unique namespace for our resources.

Schema Validation: The openAPIV3Schema section defines what fields are allowed and their validation rules. Notice how we use enum to restrict database types and pattern to validate version formats.

Required Fields: We specify that type, version, and storage are mandatory fields.

Get Chitrangna Bhatt’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Apply this CRD to your cluster:

kubectl apply -f database-crd.yaml
customresourcedefinition.apiextensions.k8s.io/databases.database.company.com created

Verify it was created:

kubectl get crd 
kubectl api-resources | grep database

CRD created

Step 2: Understanding Controllers and Operators

Before we create actual database instances, we need to understand how they’ll be managed automatically.

What is a Controller?

A controller is a program that watches for changes to specific Kubernetes resources and takes action to maintain the desired state (you can check this with spec in yaml). It follows this simple loop:

1. Watch: Monitor for changes to resources
2. Compare: Check current state vs desired state
3. Act: Take action to reconcile differences
4. Repeat: Continue monitoring

For our Database resource, a controller would:

• Watch for new Database resources being created
• Create the actual PostgreSQL/MySQL pods and services
• Monitor database health and update status
• Handle updates and deletions

You can relate this to how replicaset/deployment maintains the number of replicas you define. And if any pod is unavailable, it’ll spin up another one to maintain the desired status. From Kubernetes documentation itself —

Kubernetes comes with a set of built-in controllers that run inside the kube-controller-manager. These built-in controllers provide important core behaviors.

The Deployment controller and Job controller are examples of controllers that come as part of Kubernetes itself (“built-in” controllers). Kubernetes lets you run a resilient control plane, so that if any of the built-in controllers were to fail, another part of the control plane will take over the work.

What is an Operator?

An operator is a controller that includes domain-specific knowledge. While a generic controller might just create pods, an operator understands the application it’s managing. So this is for tasks that are more complex and application specific than what controller can manage. For example upgrading a database or recovering from backup. So yes, all operators are controllers.

Step 3: Creating Your First Database Resource

Now let’s create a database using our CRD:

apiVersion: database.company.com/v1
kind: Database
metadata:
  name: user-service-postgres
  namespace: development
  labels:
    app: user-service
    environment: dev
spec:
  type: postgresql
  version: "13"
  storage: 20Gi

Apply this resource:

kubectl apply -f user-service-db.yaml
database.database.company.com/user-service-postgres created

Check if it was created:

kubectl get databases -n development
kubectl describe database user-service-postgres -n development

Database created using CRD in background

You’ll notice the resource exists, but nothing actually happens yet — that’s because we don’t have a controller running to act on it. Kubernetes does not yet understand what should it relate your CRD’s specs (type, version, storage) to.

Step 4: Controller

Writing a controller is out of scope for this article, but you can check out how to do that here in case you’re interested. It would basically listen to events on CRD and take actions accordingly.

Step 5: What the Controller Would Do

When we eventually deploy a controller, here are all the resources that can be created automatically when you create a Database resource. Also heads up, you will need to create development namespace before these.

Resources we’ll be creating

1. Create Database Deployment

The controller can generate a deployment like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-postgres
  namespace: development
spec:
  replicas: 1
  selector:
    matchLabels:
      app: user-service-postgres
  template:
    metadata:
      labels:
        app: user-service-postgres
    spec:
      containers:
      - name: postgres
        image: postgres:13
        env:
        - name: POSTGRES_DB
          value: user-service-postgres
        - name: POSTGRES_USER
          value: dbuser
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: user-service-postgres-credentials
              key: password
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 1
            memory: 2Gi
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: user-service-postgres-pvc

2. Create Persistent Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: user-service-postgres-pvc
  namespace: development
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  # storageClassName: fast-ssd - This is how you can specify storage class type according to what your system supports

3. Create Database Service

apiVersion: v1
kind: Service
metadata:
  name: user-service-postgres
  namespace: development
spec:
  selector:
    app: user-service-postgres
  ports:
  - port: 5432
    targetPort: 5432
  type: ClusterIP

4. Create Credentials Secret

apiVersion: v1
kind: Secret
metadata:
  name: user-service-postgres-credentials
  namespace: development
type: Opaque
data:
  username: ZGJ1c2Vy  # base64 encoded "dbuser"
  password: <randomly-generated-password in base64 encoded>
  database: dXNlci1zZXJ2aWNlLXBvc3RncmVz  # base64 encoded database name

5. Update Resource Status

You could also use controller to update the Database resource status like below (CRD will need to be updated accordingly):

status:
  phase: Ready
  endpoint: user-service-postgres.development.svc.cluster.local:5432
  credentials: user-service-postgres-credentials

Step 6: Advanced CRD Features

Schema Validation in Action

Let’s test our validation by trying to create an invalid database:

apiVersion: database.company.com/v1
kind: Database
metadata:
  name: invalid-db
spec:
  type: invalid-engine  # This should fail
  version: "not-a-version"  # This should also fail
  storage: invalid-size  # This too

When you apply this, Kubernetes will reject it:

kubectl apply -f invalid-db.yaml
# Error: validation failure in databases.database.company.com

Error when applying wrong configuration

Using Short Names

Thanks to our shortNames configuration, you can use shortcuts:

# These are equivalent
kubectl get databases
kubectl get db

Status Subresource

The status section is special — it can only be updated by controllers, not by regular users. This prevents users from manually changing the database status to “Ready” when it’s actually failing.

Step 7: Real life Example

MySQL Database:

apiVersion: database.company.com/v1
kind: Database
metadata:
  name: inventory-mysql
  namespace: production
spec:
  type: mysql
  version: "8.0"
  storage: 50Gi

Step 8: Deletion of database

You can use the following command to delete the database —

kubectl delete database user-service-postgres -n development

Deleting database will not delete pvc, you will need to delete this separately, and other resources you might have created manually.

Common Pitfalls and How to Avoid Them

CRD Validation Issues

# Always test your CRD schema before deploying
kubectl apply --dry-run=server -f database-crd.yaml

Version Management

Always plan for CRD versioning from the beginning:

versions:
- name: v1
  served: true
  storage: true
- name: v1beta1
  served: true
  storage: false  # Deprecated version

Conclusion —

I’m as relieved reaching here as you are ;)

Custom Resource Definitions provide a powerful foundation for managing complex infrastructure in Kubernetes. By treating databases as native Kubernetes resources, we eliminate manual effort and errors.

The key insight is that CRDs are just the beginning — they define the “what” (database specifications), while controllers and operators provide the “how” (actual database management). Start with well-designed CRDs, then gradually add automation as your team’s expertise grows.

What challenges are you solving with Custom Resource Definitions?