Techniques
This page outlines some of the techniques that may not be obvious but allow you to extract more power from Graphile Worker.
Limiting concurrency
Out of the box, Graphile Worker supports jobs being added to a named queue (in which case that queue has a concurrency of 1) or not being added to a queue (in which case the jobs can execute as quickly as there are workers to execute them). Sometimes you want to limit concurrency to a small number, but larger than 1... Here are some techniques you can use to do this.
Dedicated worker with specific concurrency
With this technique, you'd make sure that only one worker supports this task identifier (and that worker supports this task identifier only), and then you run this worker with the desired concurrency and make sure not to add the jobs to a named queue.
This is effective, but it requires running a separate worker which may spend a lot of time idle.
A variant of this technique is to run N dedicated workers each with concurrency M, where N*M is your desired concurrency.
Multiple queue names
Whether via round-robin, random, or some other method; this technique has you create as many named queues as you need concurrency and schedule tasks into these queues.
One trade-off of this approach is that jobs may not be ran in the desired order - order is only maintained within each named queue, but if jobs in one named queue execute faster or slower than others, the tasks may come out of sync.
Forbidden flags
Managing priority
So you've got some new jobs that need to execute right now, but your workers are all busy executing long-running boring background tasks already? Here's some solutions!
Dedicated high-priority worker
Assuming that your high priority tasks belong to their own task identifier, you can run a separate worker that's dedicated to this task identifier and will pick up on the jobs as soon as they're available (assuming you have sufficient concurrency).
Run greater concurrency
Either running more workers or have your workers have higher concurrency, or both.
Limit concurrency of slow tasks
Use the "limiting concurrency" techniques above on your slow tasks, which should mean that you always have reserve capacity available for high priority jobs.
Scale up on-demand
Worker events
Use the events system to detect when your workers are
overwhelmed, and scale up as necessary. One common signal for this is when the
run_at
of a job that you start executing is significantly before the current
time minus the pollInterval
(assuming that your database and worker clocks are
synchronized).