A quick introduction to Redis Queue (RQ) to process long-running jobs

Photo by Will Truettner on Unsplash

When building an application, there are times when you need to submit a long-running job to the backend and compute the results before returning it back to the front end. Some examples include image processing tasks, video uploads, and sending bulk emails to the recipients. From the client's perspective, these are very annoying when you have to look at the endless spinning spinners to wait for the requests to be completed, and eventually, see the requests timed out in the end.

There are multiple ways of approaching this problem, including my previous post on using GCP Cloud Tasks to manage and schedule jobs. Celery is another popular library that can be used to manage background tasks in Python applications. If you are looking at simpler and more intuitive APIs, Redis Queue (RQ) will be a good choice to perform the task.

Redis Queue (RQ)

RQ is built on top of Redis, a popular in-memory data store that provides fast, reliable storage for your data. RQ provides a messaging queue that allows you to store and retrieve messages in a timely manner, making it ideal for processing background jobs. By using a separate queue to store and manage the background tasks, this helps in separating the areas of concern, i.e. the main application and the functions that process the…