How to Upgrade Debezium in Docker: A Complete Step-by-Step Guide

Debezium Upgrade

Introduction

If you’re running Debezium for Change Data Capture (CDC) in a Docker environment and suddenly encounter the dreaded MySQL syntax error “check the manual that corresponds to your MySQL server version for the right syntax to use near ‘MASTER STATUS’”, you’re not alone. This error typically appears when using older Debezium versions with MySQL 8.0.23 or newer.

In this comprehensive guide, I’ll walk you through upgrading Debezium from version 2.7 to the latest 3.6.1.Final release, based on a real-world production upgrade scenario.

Why Upgrade? The Problem We Faced
Our production Kafka Connect cluster running Debezium started throwing errors:

java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; 
check the manual that corresponds to your MySQL server version for 
the right syntax to use near 'MASTER STATUS' at line 1

The root cause? MySQL deprecated the SHOW MASTER STATUS command in version 8.0.23 and replaced it with SHOW BINARY LOG STATUS. Older Debezium versions (pre-1.9) don’t support this new syntax.

The solution? Upgrade to Debezium 3.6, which fully supports modern MySQL versions and includes tons of performance improvements and new features.

What You’ll Need
Before diving in, ensure you have:

✅ Docker and docker-compose installed
✅ Access to Kafka Connect REST API (default: http://localhost:8083)
✅ jq for JSON parsing (optional but highly recommended)
✅ SSH/terminal access to your Docker host
✅ Backup strategy for your data volumes

Understanding Your Setup
Step 1: Identify Your Current Configuration
First, let’s see what we’re working with:

# List all running containers
docker ps -a

# Check your current Debezium version
curl http://localhost:8083/connector-plugins | jq -r '.[] | select(.class | contains("debezium")) | "\(.class): \(.version)"'

In our case, the output showed:

io.debezium.connector.mysql.MySqlConnector: 2.7.0.Final

Step 2: Locate Your Project Files
The typical Debezium Docker setup looks like this:

project-root/
├── docker-compose.yml          # Orchestration file
├── junk/                       # Or your working directory
│   └── docker-compose.yml
├── debezium/                   # Custom Debezium build
│   ├── Dockerfile
│   ├── config.yml              # Prometheus metrics config
│   └── connectors/             # Additional connector JARs
└── docker_data/                # Persistent volumes

Use these commands to find your files:

# Find docker-compose files
find ~ -name "docker-compose.yml" 2>/dev/null | grep -i debezium

The Upgrade Process: 10 Essential Steps

Step 1: Backup Everything (Don’t Skip This!)

Critical first step: Always backup before making changes.

# Backup all connector configurations
curl http://localhost:8083/connectors > ~/all_connectors_backup.json

# Backup each connector individually
for connector in $(curl -s http://localhost:8083/connectors | jq -r '.[]'); do
  curl -s http://localhost:8083/connectors/$connector > ~/backup_${connector}.json
  echo "✓ Backed up: $connector"
done

# Create a timestamped backup of data volumes
sudo tar -czf ~/debezium_data_backup_$(date +%Y%m%d).tar.gz /path/to/docker_data/

# Backup your Dockerfile
cd debezium/
cp Dockerfile Dockerfile.backup

Pro tip: You can also create a Docker image backup of your current container:

docker commit <your-connect-container-name> debezium-connect-backup:2.7

Step 2: Review Your Dockerfile

Our Dockerfile looked like this:

ARG DEBEZIUM_VERSION
FROM quay.io/debezium/connect:${DEBEZIUM_VERSION}

ARG JMX_AGENT_VERSION
RUN mkdir /kafka/etc && cd /kafka/etc &&\
    curl -so jmx_prometheus_javaagent.jar \
    https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/$JMX_AGENT_VERSION/jmx_prometheus_javaagent-$JMX_AGENT_VERSION.jar

COPY connectors/confluentinc-kafka-connect-avro-converter/lib/*.jar /kafka/connect/kafka-connect-avro-converter/
COPY connectors/confluentinc-kafka-connect-s3/lib/*.jar /kafka/connect/kafka-connect-s3/
COPY connectors/confluentinc-kafka-connect-jdbc/lib/*.jar /kafka/connect/kafka-connect-jdbc/
COPY connectors/confluentinc-kafka-connect-json-schema-converter/lib/*.jar /kafka/connect/confluentinc-kafka-connect-json-schema-converter/

COPY config.yml /kafka/etc/config.yml

USER root
RUN update-crypto-policies --set LEGACY

Step 3: Fix Dockerfile Compatibility Issues
The problem: The update-crypto-policies command doesn’t exist in newer Debezium 3.6 images and will cause build failures.

The fix: Comment it out or remove it:

cd /path/to/debezium/
sed -i 's/^RUN update-crypto-policies --set LEGACY$/# RUN update-crypto-policies --set LEGACY/' Dockerfile

Your updated Dockerfile should now end with:

USER root
# RUN update-crypto-policies --set LEGACY  # Not available in Debezium 3.6+

Why this works: Debezium 3.6 handles TLS/SSL compatibility differently and doesn’t require legacy crypto policies.

Step 4: Set the New Debezium Version
You have two options:

Option A: Using a .env file (Recommended for persistence)

cd /home/amitk/Documents/cdc_upgrade/

# Create or update .env file
echo "DEBEZIUM_VERSION=3.6" > .env

# Verify
cat .env

Option B: Export as environment variable (Session-only)

export DEBEZIUM_VERSION=3.6
echo $DEBEZIUM_VERSION

Step 5: Stop the Debezium Connect Container

Important: We’re only stopping the Connect container, not the entire stack. This keeps Kafka, Zookeeper, Schema Registry, and other services running.

# Check what's running
docker ps

# Stop only Connect
docker stop debezium-datalake-connect-1

# Verify it's stopped
docker ps -a | grep connect

Your output should show something like:

CONTAINER ID   IMAGE          STATUS
11803b81c614   ...            Exited (0) 2 seconds ago

Step 6: Rebuild with the New Version
Navigate to your project root and rebuild:

cd /home/amitk/Documents/cdc_upgrade/

# Rebuild with no cache (ensures fresh build)
docker-compose -f junk/docker-compose.yml build --no-cache connect

What’s happening behind the scenes:

1. Docker pulls quay.io/debezium/connect:3.6
2. Downloads JMX Prometheus agent
3. Copies your custom connector JARs
4. Builds the new image

You’ll see output like:

[+] Building 45.2s (12/12) FINISHED
 => [1/8] FROM quay.io/debezium/connect:3.6
 => [2/8] RUN mkdir /kafka/etc && cd /kafka/etc...
 => [8/8] COPY config.yml /kafka/etc/config.yml
Successfully tagged debezium-upgrade-datalake-connect-1:latest

Step 7: Remove Old Container and Start Fresh

# Remove the old stopped container
docker rm debezium-upgrade-datalake-connect

# Start the new upgraded container
docker-compose -f junk/docker-compose.yml up -d connect

# Watch the logs
docker logs -f debezium-upgrade-datalake-connect-1

What to look for in logs:

INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect)
INFO Debezium version: 3.6.1.Final

Step 8: Wait for Connect to Fully Start

Kafka Connect needs time to initialize. Typically 30–60 seconds.

# Wait for startup
sleep 45

# Check if Connect API is responding
curl http://localhost:8083/

You should see JSON output with Connect’s version info.

Step 9: Verify the Upgrade

Now for the moment of truth:

# Check Debezium version via REST API
curl -s http://localhost:8083/connector-plugins | \
  jq -r '.[] | select(.class | contains("debezium")) | "\(.class): \(.version)"'

Expected output:

io.debezium.connector.mysql.MySqlConnector: 3.6.1.Final
io.debezium.connector.postgresql.PostgresConnector: 3.6.1.Final
io.debezium.connector.mongodb.MongoDbConnector: 3.6.1.Final

You can also verify by checking JAR files inside the container:

docker exec hdfccredila-analytics-datalake-connect-1 \
  find /kafka/connect -name "debezium-connector-mysql*.jar" -exec basename {} \;

# Output: debezium-connector-mysql-3.6.1.Final.jar

Step 10: Check Connector Status
The moment of truth — did our connectors survive the upgrade?

# List all connectors
curl http://localhost:8083/connectors

# Check status of all connectors
curl -s http://localhost:8083/connectors | jq -r '.[]' | while read connector; do
  state=$(curl -s http://localhost:8083/connectors/$connector/status | jq -r '.connector.state')
  tasks=$(curl -s http://localhost:8083/connectors/$connector/status | jq -r '.tasks | length')
  echo "  $connector: $state ($tasks tasks)"
done

Possible outcomes:

✅ RUNNING — Perfect! No action needed.

⚠️ FAILED — Restart the connector:

curl -X POST http://localhost:8083/connectors/<connector-name>/restart
sleep 10
curl http://localhost:8083/connectors/<connector-name>/status | jq '.'

❌ Not found — Recreate from backup:

curl -X POST -H "Content-Type: application/json" \
  --data @~/backup_<connector-name>.json \
  http://localhost:8083/connectors/

What’s New in Debezium 3.6?
Now that you’ve upgraded, here’s what you gained:

Get Amit Kumar Manjhi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Performance Improvements

• Reduced CPU overhead in MySQL connector
• Better memory management for table history/schemas
• Optimized schema caching

New Features

• Debezium Platform monitoring support
• MySQL GTID tags support
• YashanDB connector (new database support)
• JDBC sink improvements for Debezium Serve
• Debezium CLI — Command-line interface for connector management

Technical Enhancements

• MySQL 8.4.x and 9.x support
• PostgreSQL 17 and 18 support
• TLS customization for CockroachDB
• Kafka 4.3 support
• Java 17+ required (Java 21+ for Server/Operator)

Bug Fixes (3.6.1.Final)

• PostgreSQL BC date parsing
• MySQL INTEGER UNSIGNED mapping
• Oracle RowId corruption fixes
• Credential leakage prevention

Troubleshooting Common Issues
Issue 1: Build Fails with “update-crypto-policies: command not found”
Error:

/bin/sh: line 1: update-crypto-policies: command not found

Solution:
Comment out or remove the line from your Dockerfile:

sed -i 's/^RUN update-crypto-policies --set LEGACY$/# RUN update-crypto-policies --set LEGACY/' Dockerfile

Issue 2: DOCKER_HOST Connection Error

Error:

docker.errors.DockerException: Error while fetching server API version: 
Not supported URL scheme http+docker

Solution:
Reset your Docker connection settings:

unset DOCKER_HOST
export DOCKER_HOST=unix:///var/run/docker.sock
docker context use default

Issue 3: Connectors in FAILED State

Check the error:

curl http://localhost:8083/connectors/<connector-name>/status | jq '.tasks[].trace'

Common fixes:

• Restart the connector: curl -X POST http://localhost:8083/connectors/<name>/restart
• Check database permissions
• Verify MySQL binary logging is enabled
• Ensure Kafka topics are accessible

Issue 4: “DEBEZIUM_VERSION variable is not set”

Error:

WARNING: The DEBEZIUM_VERSION variable is not set. Defaulting to a blank string.

Solution:

export DEBEZIUM_VERSION=3.6
# Or create .env file
echo "DEBEZIUM_VERSION=3.6" > .env

Complete Automated Upgrade Script

For those who want to automate the entire process, here’s a complete bash script:

#!/bin/bash
set -e  # Exit on error

# Configuration - MODIFY THESE
PROJECT_ROOT="/home/amitk/Documents/debezium-upgrade"
DEBEZIUM_FOLDER="debezium"
COMPOSE_FILE="junk/docker-compose.yml"
NEW_VERSION="3.6"
CONNECT_API="http://localhost:8083"

echo "🚀 Debezium Upgrade Script - Version $NEW_VERSION"
echo "================================================"

cd $PROJECT_ROOT

# Step 1: Backup
echo ""
echo "📦 Step 1/10: Backing up connectors..."
mkdir -p ~/debezium-backups/$(date +%Y%m%d)
for connector in $(curl -s $CONNECT_API/connectors | jq -r '.[]'); do
  curl -s $CONNECT_API/connectors/$connector > ~/debezium-backups/$(date +%Y%m%d)/backup_${connector}.json
  echo "  ✓ Backed up: $connector"
done

# Step 2: Backup Dockerfile
echo ""
echo "📦 Step 2/10: Backing up Dockerfile..."
cp $DEBEZIUM_FOLDER/Dockerfile $DEBEZIUM_FOLDER/Dockerfile.backup.$(date +%Y%m%d)

# Step 3: Fix Dockerfile
echo ""
echo "🔧 Step 3/10: Fixing Dockerfile compatibility..."
sed -i 's/^RUN update-crypto-policies --set LEGACY$/# RUN update-crypto-policies --set LEGACY/' $DEBEZIUM_FOLDER/Dockerfile
echo "  ✓ Dockerfile updated"

# Step 4: Set version
echo ""
echo "🔢 Step 4/10: Setting Debezium version to $NEW_VERSION..."
export DEBEZIUM_VERSION=$NEW_VERSION
echo "DEBEZIUM_VERSION=$NEW_VERSION" > .env
echo "  ✓ Version set to $NEW_VERSION"

# Step 5: Stop container
echo ""
echo "🛑 Step 5/10: Stopping Connect container..."
CONTAINER_NAME=$(docker-compose -f $COMPOSE_FILE ps -q connect | xargs docker inspect -f '{{.Name}}' | sed 's/\///' || echo "not-found")
if [ "$CONTAINER_NAME" != "not-found" ]; then
  docker stop $CONTAINER_NAME
  echo "  ✓ Stopped: $CONTAINER_NAME"
fi

# Step 6: Rebuild
echo ""
echo "🔨 Step 6/10: Rebuilding with new version (this may take a few minutes)..."
docker-compose -f $COMPOSE_FILE build --no-cache connect

# Step 7: Remove old container
echo ""
echo "🗑️  Step 7/10: Removing old container..."
if [ "$CONTAINER_NAME" != "not-found" ]; then
  docker rm $CONTAINER_NAME
  echo "  ✓ Removed: $CONTAINER_NAME"
fi

# Step 8: Start new container
echo ""
echo "🚀 Step 8/10: Starting upgraded Connect container..."
docker-compose -f $COMPOSE_FILE up -d connect

# Step 9: Wait for startup
echo ""
echo "⏳ Step 9/10: Waiting for Connect to start (60 seconds)..."
for i in {1..12}; do
  sleep 5
  if curl -s $CONNECT_API/ > /dev/null 2>&1; then
    echo "  ✓ Connect is ready!"
    break
  fi
  echo "  ... still waiting ($((i*5))s)"
done

# Step 10: Verify
echo ""
echo "✅ Step 10/10: Verifying upgrade..."
echo ""
echo "Debezium Version:"
curl -s $CONNECT_API/connector-plugins | jq -r '.[] | select(.class | contains("debezium")) | "  \(.class): \(.version)"'

echo ""
echo "Connector Status:"
curl -s $CONNECT_API/connectors | jq -r '.[]' | while read connector; do
  state=$(curl -s $CONNECT_API/connectors/$connector/status | jq -r '.connector.state')
  tasks=$(curl -s $CONNECT_API/connectors/$connector/status | jq -r '.tasks | length')
  if [ "$state" = "RUNNING" ]; then
    echo "  ✓ $connector: $state ($tasks tasks)"
  else
    echo "  ⚠️  $connector: $state ($tasks tasks)"
  fi
done

echo ""
echo "================================================"
echo "🎉 Upgrade complete!"
echo ""
echo "Backups stored in: ~/debezium-backups/$(date +%Y%m%d)/"
echo "Dockerfile backup: $PROJECT_ROOT/$DEBEZIUM_FOLDER/Dockerfile.backup.$(date +%Y%m%d)"
echo ""
echo "Next steps:"
echo "  - Monitor connector logs: docker logs -f <connect-container-name>"
echo "  - Check connector status: curl $CONNECT_API/connectors/<name>/status"
echo "  - Restart failed connectors: curl -X POST $CONNECT_API/connectors/<name>/restart"

Save this as upgrade-debezium.sh, make it executable, and run:

chmod +x upgrade-debezium.sh
./upgrade-debezium.sh

Best Practices & Pro Tips
1. Test in Lower Environments First
Never upgrade production directly. Test in dev/staging environments first.

2. Schedule During Low-Traffic Windows
Plan your upgrade during maintenance windows or low-traffic periods.

3. Monitor After Upgrade
Watch these metrics post-upgrade:

• Connector lag (kafka-consumer-groups — describe)
• CPU and memory usage
• Kafka Connect JMX metrics (if using Prometheus/Grafana)

4. Keep Backups for 30 Days
Don’t delete backups immediately. Keep them for at least 30 days.

5. Use Version Pinning
Always specify exact versions rather than using latest:

# Good
DEBEZIUM_VERSION=3.6.1.Final

# Avoid
DEBEZIUM_VERSION=latest

6. Read Release Notes
Always review Debezium release notes before upgrading for breaking changes.

Rollback Plan (Just in Case)

If something goes wrong, here’s how to rollback:

# Option 1: Use backup image
docker stop <new-container>
docker run -d --name connect-rollback debezium-connect-backup:2.7

# Option 2: Rebuild with old version
export DEBEZIUM_VERSION=2.7
docker-compose build --no-cache connect
docker-compose up -d connect

# Option 3: Restore from Dockerfile backup
cp Dockerfile.backup Dockerfile
docker-compose build --no-cache connect
docker-compose up -d connect

Version Compatibility Matrix

Monitoring Your Upgraded System

After upgrading, keep an eye on these key metrics:

Kafka Connect Metrics (JMX)

# If using Prometheus (via JMX exporter on port 8080)
curl http://localhost:8080/metrics | grep debezium

Key metrics to watch:

• debezium_metrics_MilliSecondsBehindSource — Replication lag
• debezium_metrics_NumberOfEventsFiltered — Event filtering
• debezium_metrics_RowsScanned — Snapshot progress
• kafka_connect_worker_metrics_task_count — Active tasks

Connector Health Check Script

#!/bin/bash
# save as check-connectors.sh

API="http://localhost:8083"
echo "Debezium Connector Health Check"
echo "================================"

for connector in $(curl -s $API/connectors | jq -r '.[]'); do
  status=$(curl -s $API/connectors/$connector/status)
  state=$(echo $status | jq -r '.connector.state')
  tasks=$(echo $status | jq -r '.tasks[].state')
  
  echo ""
  echo "Connector: $connector"
  echo "  State: $state"
  echo "  Tasks:"
  echo "$tasks" | while read task_state; do
    echo "    - $task_state"
  done
  
  # Check for errors
  errors=$(echo $status | jq -r '.tasks[].trace // empty')
  if [ ! -z "$errors" ]; then
    echo "  ⚠️  ERRORS:"
    echo "$errors" | head -5
  fi
done

Conclusion
Upgrading Debezium from 2.7 to 3.6 in Docker is straightforward when you follow these steps:

✅ Backup everything — connectors, configurations, data volumes
✅ Fix Dockerfile — remove incompatible commands
✅ Set new version — via .env or export
✅ Rebuild and restart — only the Connect service
✅ Verify and monitor — ensure connectors are running

Key takeaways:

• Always test in non-production first
• Backups are your safety net
• Monitor for 24–48 hours post-upgrade
• Keep rollback options ready
• Read release notes for breaking changes

What you gained:

✨ Support for modern MySQL versions (8.4.x, 9.x)
⚡ Better performance and lower CPU usage
🎯 New features like Debezium CLI and Platform
🐛 Important bug fixes and security patches
🔮 Future-proof compatibility

Additional Resources
📚 Debezium 3.6 Documentation
📝 Debezium 3.6 Release Notes
💬 Debezium Community Chat
🐛 Report Issues
📺 Debezium YouTube Channel

About the Author

This guide was created based on a real production upgrade scenario in August 2026. If you found it helpful, consider sharing it with your team!

Questions or issues? Drop a comment below or reach out on the Debezium community chat.

Happy streaming! 🚀