Distributed lock

Lets see how pods run the schedular

Basically in the App, as we all know main will initiate/start all the config , services and Kafka producers and schedulers.

Assume we have a service which fetch data from external API Every 5 mins and publish to Kafka event

And created a scheduler and assigned a job of fetching data in every 5 mins

Lets say we are deploying this application in Kubernetes and running instance in 3 pods

Application is running on the 3 pods , the schedular job runs once in 5 mins not once per pod .

pod1- starts schedular

pod2 — starts schedular

pod3 starts schedular — now all of my 3 pods try to run the same job every 5 mins

why because pod knows only its own memory ,timer and process. It has no idea about other pods process

Get kiran vemula’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If the application running in one pod, it works fine (pod 1 starts and run schedular job every 5 mins)

its creates duplicate api calls and duplicate events. We need to ensure only 1 pod executes the job at a time

Solution : shared lock/distributed lock

1. All three pods starts the schedular during application start-up — means all three pods initialize the schedular instance
2. But Schedular job is protected by the Distributed lock — all the pods host the schedular but one pod executes the scheduler job
   every pod is allowded to initilize the schedular
   before running the job , the scheduler acquires the shared lock which is stored in shared database

only the pod acquires lock can run the job

this give us the leader style execution without implementing the separate leader election components
out of my 3 pods, 1 pod acquires a lock and runs the job, other 2 remains do nothing for that tick

Important Note in this approach
scheduler is not just a in memory cron, it should be configured with the DB backed distributed lock which is shared by all pods , means schedular decision should be coordinated across the replicas

this design only works if all pods works same database and same lock

Trade-offs to keep in mind
1.Db becomes the part of scheduling, if DB is down, the schedular may not able to co-ordinate safely

2. Lock behaviour should be monitored — Need to have much more logs to see