Laravel's queue system transforms time-consuming tasks into responsive background processes, ensuring applications remain fast and scalable. Laravel's queue system allows you to defer these long-running tasks to run in an ordered or controlled parallel manner, ensuring the server remains responsive and users do not have to wait. This guide covers comprehensive queue management strategies for production environments.
Queue Fundamentals and Configuration
Queue Driver Selection
Configure queue connections in config/queue.php for production reliability:
// .env configuration
QUEUE_CONNECTION=redis // Recommended for production
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
Redis provides superior performance and reliability compared to database queues. For high-volume applications, consider managed services like AWS ElastiCache or Redis Cloud.
Creating Background Jobs
Generate job classes using Artisan commands:
php artisan make:job ProcessPayment
// Generated job class
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
class ProcessPayment implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $timeout = 120;
public $tries = 3;
public function __construct(private $payment) {}
public function handle()
{
// Process payment logic
$this->payment->process();
}
public function failed($exception)
{
// Handle job failure
Log::error('Payment processing failed: ' . $exception->getMessage());
}
}
Production Queue Workers
Supervisor Configuration
For production, you need a way to keep your queue:work processes running. We need a program that supervises the worker process, restarts it if it fails, and potentially scales the number of processes.
Create supervisor configuration for reliable worker management:
# /etc/supervisor/conf.d/laravel-worker.conf
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work redis --sleep=3 --tries=3 --max-time=3600 --max-jobs=1000
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/worker.log
--max-jobs tells Laravel that this worker can only process 1000 jobs. After it reaches the limit it'll be shut down. Then memory will be freed up and supervisor restarts the worker. --max-time tells Laravel that this worker can only live for an hour.
Supervisor Management Commands
# Update supervisor configuration
sudo supervisorctl reread
sudo supervisorctl update
# Start workers
sudo supervisorctl start laravel-worker:*
# Monitor worker status
sudo supervisorctl status
# Restart workers during deployment
sudo supervisorctl restart laravel-worker:*
Queue Priority and Multiple Queues
Queue Prioritization Strategy
Since payments are the most important jobs it's probably a good idea to separate them and handle them with priority. The same can be true for notifications as well.
// Job dispatching with priorities
ProcessPayment::dispatch($payment)->onQueue('payments');
SendNotification::dispatch($user)->onQueue('notifications');
GenerateReport::dispatch($data)->onQueue('default');
Worker Configuration for Priorities
# High-priority workers (payments first)
php artisan queue:work --queue=payments,notifications,default
# Notification workers
php artisan queue:work --queue=notifications,default
# General workers
php artisan queue:work --queue=default
Configure different supervisor groups for each priority:
[program:payment-worker]
command=php /var/www/html/artisan queue:work --queue=payments,notifications,default
numprocs=2
[program:notification-worker]
command=php /var/www/html/artisan queue:work --queue=notifications,default
numprocs=3
[program:default-worker]
command=php /var/www/html/artisan queue:work --queue=default
numprocs=2
Laravel Horizon Implementation
Horizon Installation and Configuration
For Redis-powered applications, Laravel Horizon provides elegant queue management:
composer require laravel/horizon
php artisan horizon:install
php artisan vendor:publish --provider="Laravel\Horizon\HorizonServiceProvider"
Configure production environment in config/horizon.php:
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['payments', 'notifications', 'default'],
'balance' => 'auto',
'maxProcesses' => 10,
'minProcesses' => 1,
'tries' => 3,
'timeout' => 60,
],
],
],
The auto strategy, which is the configuration file's default, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your notifications queue has 1,000 pending jobs while your render queue is empty, Horizon will allocate more workers to your notifications queue until the queue is empty.
Horizon Supervisor Configuration
# /etc/supervisor/conf.d/horizon.conf
[program:horizon]
process_name=%(program_name)s
command=php /var/www/html/artisan horizon
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/horizon.log
stopwaitsecs=3600
Memory Management and Performance
Worker Memory Optimization
Queued jobs can cause some memory leaks. Unfortunately, I don't know the exact reasons but not everything is detected by PHP's garbage collector. As time goes, and your worker processes more jobs it uses more and more memory.
Implement memory management strategies:
# Memory-conscious worker configuration
php artisan queue:work --max-jobs=1000 --max-time=3600 --memory=512
# Monitor memory usage
php artisan queue:monitor redis:default,redis:payments --max=100
Job Timeout Configuration
Configure appropriate timeouts in your job classes:
class ProcessLargeFile implements ShouldQueue
{
public $timeout = 300; // 5 minutes
public $tries = 2;
public $maxExceptions = 1;
public function retryAfter()
{
return 60; // Retry after 1 minute
}
}
Production Deployment Integration
cPanel Queue Setup
Based on your deployment notes, configure queues in cPanel environments:
# Set up queue worker for background processing
cd /home/myLaravelApp/public_html
# Process queued jobs
/opt/cpanel/ea-php82/root/usr/bin/php /home/myLaravelApp/public_html/artisan queue:work --max-time=60 --sleep=3 --tries=3 --max-jobs=50
# Add to cron for automatic restart
# */5 * * * * /opt/cpanel/ea-php82/root/usr/bin/php /home/myLaravelApp/public_html/artisan queue:restart
Deployment Automation
Create deployment scripts that handle queue workers:
#!/bin/bash
# deployment/deploy.sh
# Stop existing workers
php artisan queue:restart
# Clear and cache optimizations
php artisan config:cache
php artisan route:cache
# Restart supervisor workers
sudo supervisorctl restart laravel-worker:*
# Start Horizon (if using)
php artisan horizon:terminate
sleep 5
php artisan horizon
Job Batching and Advanced Features
Batch Processing
Handle large datasets efficiently with job batching:
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;
$batch = Bus::batch([
new ProcessUser($user1),
new ProcessUser($user2),
new ProcessUser($user3),
])->then(function (Batch $batch) {
// All jobs completed successfully
})->catch(function (Batch $batch, Throwable $e) {
// Handle batch failure
})->finally(function (Batch $batch) {
// Cleanup operations
})->dispatch();
Unique Jobs
Prevent duplicate job processing:
class ProcessOrder implements ShouldQueue, ShouldBeUnique
{
public $uniqueFor = 3600; // 1 hour
public function uniqueId()
{
return $this->order->id;
}
}
Monitoring and Maintenance
Queue Health Monitoring
Implement comprehensive monitoring:
# Check queue status
php artisan queue:monitor redis:default --max=100
# View failed jobs
php artisan queue:failed
# Retry failed jobs
php artisan queue:retry all
# Clear failed jobs
php artisan queue:flush
Horizon Dashboard Access
Configure Horizon dashboard authentication:
// App\Providers\HorizonServiceProvider
public function gate()
{
Gate::define('viewHorizon', function ($user) {
return in_array($user->email, [
'admin@yourdomain.com',
]);
});
}
Performance Optimization
Redis Configuration
Optimize Redis for queue performance:
# redis.conf optimizations
maxmemory 2gb
maxmemory-policy allkeys-lru
save 900 1
save 300 10
save 60 10000
Worker Scaling Strategy
The default production environment is configured to start a maximum of 10 worker processes and automatically balance the number of worker processes assigned to each queue.
Scale workers based on load:
// config/horizon.php
'environments' => [
'production' => [
'supervisor-1' => [
'maxProcesses' => 20,
'minProcesses' => 2,
'balanceMaxShift' => 3,
'balanceCooldown' => 3,
],
],
],
Effective queue management ensures Laravel applications handle background processing reliably and efficiently. Through proper supervisor configuration, strategic queue prioritization, and comprehensive monitoring, production environments can process thousands of jobs seamlessly while maintaining optimal performance and fault tolerance.
