[Laravel] horizon worker configuration

Efficient Job Management with Laravel Queue and Horizon

In our workflow, there are two types of jobs, both of which run on the same server:

1. Simple jobs, such as sending emails and SMS.
2. Long-running jobs that export data to CSV files, typically taking about 5 minutes or so.

To accommodate these needs, we configured two connections in our config/queue.php, each with a different retry_after value. Additionally, we use Laravel Horizon to configure and monitor the Laravel queue.

Deployment and Monitoring

The production server can dispatch either SimpleJob or LongRunningJob to the queue, and the worker server picks them up for asynchronous processing. The command to deploy Horizon on the worker server is php artisan horizon.

// Dispatch a simple job
SimpleJob::dispatch();

// Dispatch a long running job
LongRunningJob::dispatch()
    ->onConnection('redis-long-job');

// config/queue.php
'connections' => [
        ...
        'redis' => [
            'driver' => 'redis',
            'connection' => 'horizon',
            'queue' => 'default-job',
            'retry_after' => 90,
            'block_for' => null,
        ],
        'redis-long-job' => [
            'driver' => 'redis',
            'connection' => 'horizon',
            'queue' => 'long-running-job',
            'retry_after' => 7200,
            'block_for' => null,
        ],
    ],

    // config/horizon.php
    'environments' => [
         ...
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['default-job'],
                'balance' => 'auto',
                'maxProcesses' => 10,
                'balanceMaxShift' => 1,
                'balanceCooldown' => 3,
                'memory' => 128,
                'tries' => 3,
                'nice' => 0,
            ],
            'supervisor-1-long-job' => [
                'connection' => 'redis-long-job',
                'queue' => ['long-running-job'],
                'balance' => 'auto',
                'maxProcesses' => 5,
                'balanceMaxShift' => 1,
                'memory' => 128,
                'tries' => 3,
                'nice' => 0,
                'balanceCooldown' => 3,
            ],
        ],
    ],

Scaling with Business Growth

Get Debby Lee’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

As our business has grown, we’ve introduced a third type of job: long-running, CPU-intensive tasks, such as video processing. To dispatch this type of job, we created a specific queue (cpu-intensive-job). We also modified config/horizon.php to include supervisor-1-long-job, configured to handle jobs in the cpu-intensive-job queue.

// Dispatch a simple job and it would be on redis default-job queue
SimpleJob::dispatch();

// Dispatch a long running job and it would be on redis long-running-job queue
LongRunningJob::dispatch()
    ->onConnection('redis-long-job');

// Dispatch a long running CPU intensive job. 
// Since we specify the queue, it would be on cpu-intensive-job queue.
LongRunningCpuIntensiveJob::dispatch()
    ->onConnection('redis-long-job')
    ->onQueue('cpu-intensive-job');

  // config/horizon.php
  'supervisor-1-long-job' => [
      'connection' => 'redis-long-job',
      'queue' => ['long-running-job', 'cpu-intensive-job'],
      'balance' => 'auto',
      'maxProcesses' => 5,
      'balanceMaxShift' => 1,
      'memory' => 128,
      'tries' => 3,
      'nice' => 0,
      'balanceCooldown' => 3,
  ],

However, after implementation, we realized that CPU-intensive jobs consume an excessive amount of computing power. Consequently, it became apparent that allocating a distinct set of compute resources specifically for CPU-intensive jobs was necessary.

Compute Resource Configuration

We now have three distinct sets of servers for our compute resources:

• Production Servers for web and API-related work.
• CPU-intensive Worker Servers for jobs on the cpu-intensive-job queue.
• Normal Worker Servers for all other jobs.

This separation ensures efficient handling and processing of jobs based on their nature and requirements, optimizing our application’s performance and scalability.

// config/horizon.php
  'supervisor-1-long-job' => [
      'connection' => 'redis-long-job',
      'queue' => ['long-running-job'], // remove 'cpu-intensive-job' from previous change
      'balance' => 'auto',
      'maxProcesses' => 5,
      'balanceMaxShift' => 1,
      'memory' => 128,
      'tries' => 3,
      'nice' => 0,
      'balanceCooldown' => 3,
  ],

The command to deploy Horizon on the worker server is php artisan horizon:work redis-long-job --queue cpu-intensive-job.