Troubleshooting and Optimizing Laravel Horizon Post-Migration to AWS ECS
Hello, I’m Debby, a DevOps professional at a startup in Tokyo, Japan. Recently, we migrated one of our job systems from AWS EC2 to AWS ECS. During this process, we observed that the CPU utilization of our ECS services was quite high. Consequently, I was assigned the task of investigating ways to reduce the CPU load.
Our queue job system utilizes Laravel Horizon, and we use Supervisor to ensure Horizon remains operational.
- Multiple Horizon instances are running within a single ECS task.
Interestingly, we discovered that multiple processors were initialized in our Supervisor configuration for reasons unknown (see the configuration file below). If you have trouble locating the Supervisor configuration, you can refer to the Horizon dashboard to see how many processors are registered for the queue.
[program:horizon]
process_name=%(program_name)s_%(process_num)02d
command=php /home/sites/my-project/artisan horizon
autostart=true
autorestart=true
user=www-data
numprocs=2 // this line should be removed
redirect_stderr=true
stdout_logfile=/home/sites/my-project/storage/logs/horizon.log2. Unique Job
We have a cron server that queries the database to dispatch queue jobs. Upon reviewing the Horizon dashboard, we discovered that duplicate jobs were being dispatched. Since rewriting the dispatch logic would require substantial effort and resources, we decided to implement the ShouldBeUnique interface for those duplicated jobs. This ensures that even if the cron server dispatches duplicate jobs, they will be discarded.
Get Debby Lee’s stories in your inbox
Join Medium for free to get updates from this writer.
Despite implementing the above changes, we continued to observe peak CPU utilization, as shown in the graph below. The CPU utilization spikes are directly correlated with the number of jobs we dispatch.
To address this issue, we are considering two approaches:
- Reducing the
max_processsetting in the Horizon configuration to limit the number of jobs processed per Supervisor instance. - Investigating the code to understand why so many jobs are being dispatched and identifying potential optimizations.
We quickly identified that a specific job dispatch was contributing to the spike by comparing the graph for that job with the graph for the queue. Fortunately, we found that we could optimize further dispatch to reduce the number of job dispatches by 90%.
