Reliable, extensible and performant
Worker 0.16 is here — freshly updated with optional TypeScript typing for tasks,
a public jobs interface, a new plugin system, and much more besides!
The new plugin system enables ESM and CommonJS to peacefully coexist, adds
support for writing tasks in import()–capable compile–to–JS languages like
TypeScript, adds support for writing tasks in alternative languages (e.g.
Python, Ruby, Rust), and could open the door to a lot more customizability in
future! Version 0.16 also introduces support for “Worker Pro” — a plugin
available to most Sponsors which monitors running workers, releases locked jobs
from workers that exit uncleanly, helps ease migrations to new releases, and
will add more quality–of–life improvements in time.
We’ve backported a large amount of v0.16’s functionality to v0.13 and v0.15; this includes plugin and Worker Pro support to ease migration to the new version.
Thank you to each and every one of our sponsors, together you have made this massive release possible!
What’s new in Version 0.16?
Graphile Worker gets a logo!
Inspired by the Graphile heart and the PostGraphile elephant, Graphile Worker gets its new heart–shaped worker–ant logo. We admire the industriousness of rainforest leaf–cutter ants, relentlessly working away traversing miles of branches to bring food and messages back to their colony. What better animal to represent the spirit of Graphile Worker?
New Graphile Worker website!
As Graphile Worker’s featureset has been growing, managing it all through a README has become challenging, both for users and for maintenance. We’ve moved all the documentation to a new website and spent some time overhauling it with new tips and notes (and corrections). You’re on that new site right now! See the new documentation at worker.graphile.org/docs/.
Graphile Config and the new plugin system
Worker is now optionally configurable with Graphile Config. This enables a whole suite of new options and features, not the least of which is you can now share preset files across multiple projects! Graphile Config takes everything we’ve learned from the PostGraphile project relating to configuration and plugins and expands these capabilities into a generic system that Worker can leverage. The plumbing has been completed, the initial hooks are there and ready to be used by everyone to make Worker even more adaptable and powerful than ever before. If you need to do something that there’s not a hook for yet, adding a suitable one is likely straightforward… get in touch for guidance!
Native ESM support!
The first thing we did with the new plugin system was to move the code for loading a task into its own plugin, and add support for loading both CommonJS files and ESM files. Since it’s a plugin, you can replace it entirely, loading tasks in any way you see fit!
Compile–to–JS language support!
With the plugin mentioned above you can add more file extensions to your config
for any “compile–to–javascript” language that can be import()–ed — just be
sure to make the relevant "loaders" available, e.g. for native TypeScript
support you might use NODE_OPTIONS="--loader ts-node/esm" npx graphile-worker
Tasks in even more languages!
And the fun doesn't stop there! We've added another built–in plugin that allows
us to offer support for writing task executors in other languages entirely:
python, bash, even fortran; choose whichever language makes the most sense for
the task at hand. All you need to do is place an executable file in the tasks/
folder and ensure it's named with the task identifier (extensions are ignored).
abortSignal
We have added the JavaScript pattern of abortSignal — a new ability to tell
tasks to abort during shutdown. This means no more waiting for the task to
complete during a graceful shutdown, so long as your task executor remembers to
listen for the abortSignal your job can exit quickly when the worker is
shutting down, and boot up again soon thereafter on a different worker.
Opt–in TypeScript typing of tasks
Now you can take down the guard rails, and assume the job type using the new
GraphileWorker.Tasks global interface. This is not for the faint of heart, and
in fact we don’t generally recommend it! But it has been requested many times,
and now it is available to use via a public–facing declaration–mergeable
interface. Declaring a global task interface allows you to use auto–complete and
payload safety for addJob and quickAddJob, and specifies the assumed types
of the payloads of your task functions. More, including important caveats, can
be found in our documentation.
Remember the caveats of what it means to assume…
Public jobs view
This release makes it much clearer that the tables in which jobs and queues are stored are private implementation details — Graphile Worker needs to be able to change these tables to add new features or perform optimizations. For those users who look into the jobs table, even though we advise against it, we've created a public view that presents a stable interface into the queued jobs. We strongly advise against polling this, it will impact the performance of your worker, but it is now more suitable for usage in automated scripts since the interface is stable.
Worker Pro: upgrade your experience!
With thanks to each and every sponsor (past and present) for reinvesting back into open source, the growth of our community–based funding over the last six years has enabled us to spend more and more time maintaining and improving our various open source packages. These past few months we're grateful to welcome Trigger.dev and Seam as the latest in the organizations reinvesting in the software they use — thank you!
Alas, a significant amount of our maintenance and development time is still self–funded; so in an attempt to make this project more sustainable, we have introduced the proprietary Worker Pro plugin, and initially made it available to sponsors on the $25/mo tier and above.
Worker Pro adds worker tracking to Graphile Worker, making it a clean separation from the open source version which does not track the individual workers. With Worker Pro, each worker checks in with a "heartbeat" every so often, and we can use this in various ways.
For example, we have had feedback from the community that having to stop all workers in advance of updating to a new release is not ideal. With Worker Pro, before performing a breaking migration a graceful shutdown of all old workers is triggered, and further old workers are prevented from starting up. Once Worker Pro has confirmed all old workers have completed their graceful shutdowns, the migration can go ahead with significantly reduced risk of conflict. This avoids the need to “scale to zero” as all workers will communicate with each other to help the system as a whole remain consistent, and the newly running workers can gracefully take over from the old.
This new plugin also helps with crashed worker recovery: once a configurable amount of time has passed without receiving a heartbeat from a registered worker, Worker Pro can assume that the worker is no longer running (power pulled, killed, crashed, etc) and release all of its locked jobs to be re–attempted.
There are a couple more major features planned for “Worker Pro”, and once they are added it will be available via the $100 sponsorship tier. Now is a great time to sponsor Graphile and lock in this awesome 75% discount launch offer for at least 18 months!
To get access to “Worker Pro”, please reach out to Benjie or Jem on Discord or email team [at] graphile [dot] org — for the introductory offer you must ensure you have access before the price goes up!
Join our community — we run a Discord server which can be used for community support and to chat to others using the Graphile suite of developer tools. Find out how to sponsor us at graphile.org/sponsor.
Full release notes found on this site or at GitHub.
Thank you Graphile sponsors!