High-Throughput Routing Queues with Laravel Horizon and Redis
Why Database Queue Drivers Hit a Ceiling Under Load
Laravel's database queue driver is convenient for low-traffic apps, but once you dispatch thousands of outbound messages or webhook deliveries per minute, row-level locking (SELECT ... FOR UPDATE SKIP LOCKED) on the jobs table creates unnecessary I/O contention on your primary MySQL instance.
Moving queue state to Redis paired with Laravel Horizon offloads job popping and metrics tracking into sub-millisecond in-memory operations while allowing worker pools to auto-scale dynamically based on queue depth.
1. Isolating Queues by Latency and Priority
Never push bulk campaign exports onto the same queue as real-time webhook acknowledgments or OTP dispatches. Separate your workloads into dedicated supervisor pools in config/horizon.php:
'environments' => [
'production' => [
'realtime-supervisor' => [
'connection' => 'redis',
'queue' => ['critical', 'webhooks'],
'balance' => 'auto',
'minProcesses' => 4,
'maxProcesses' => 24,
'balanceMaxShift' => 4,
'balanceCooldown' => 3,
'tries' => 3,
'timeout' => 30,
],
'bulk-supervisor' => [
'connection' => 'redis',
'queue' => ['campaigns', 'exports'],
'balance' => 'simple',
'processes' => 8,
'tries' => 2,
'timeout' => 300,
],
],
],
2. Throttling Outbound API Calls with Redis::throttle()
Scaling up to 24 concurrent workers can easily overwhelm downstream carrier or CRM APIs that enforce strict per-second throughput limits. Wrap external HTTP requests in job middleware using Redis::throttle() so excess jobs release cleanly back onto the queue without burning $tries:
<?php
namespace App\Jobs\Middleware;
use Closure;
use Illuminate\Support\Facades\Redis;
class RateLimitUpstreamProvider
{
public function handle(object $job, Closure $next): void
{
Redis::throttle('provider:slooce-outbound')
->allow(50)
->every(1)
->then(
fn () => $next($job),
fn () => $job->release(2)
);
}
}
Remember to set public int $maxExceptions = 3; on your job class whenever using $job->release(). That ensures rate-limit releases do not prematurely mark a healthy job as failed while genuine unhandled exceptions still trigger failure after three attempts.
3. Exponential Backoff and Graceful Shutdown
Define an explicit backoff() schedule on jobs interacting with external networks so transient 502/504 gateway errors recover automatically:
public function backoff(): array
{
return [5, 30, 120];
}
Finally, always configure retry_after in config/queue.php to be strictly greater than your longest supervisor timeout (for example, timeout = 300 and retry_after = 360) so Redis never re-queues a long-running job while a worker is still actively executing it.
Written by Gulfaraz Arshad
Software Developer at Cruise Brains specializing in Laravel, PHP, MySQL performance optimization, and high-throughput API architectures.