Background jobs

Scheduled publishing and other work the site does later run as Payload jobs. Jobs are stored in the database, not on the container's disk, so a new container picks up where the old one stopped.

What runs as a job:

• Scheduled publishing (and unpublishing), wherever there are drafts.
• Learning from a published translation: Translation memory takes in each translation published in a language other than the default (with AI translation on).
• Translation evaluations, Proofread the site and Translate a file, which take minutes.
• Upkeep of translated files every five minutes: restarting a translation whose server stopped, and deleting files older than seven days.
• The nightly reset of a public demo, with DEMO_MODE and DEMO_RESET (see Live demo).

Who runs them #

The server itself, with nothing to set up. From the moment it starts, it runs the jobs that are due once a minute (src/jobs/start.ts). It runs them through its own run endpoint, inside a request, so a scheduled publish refreshes cached pages just as a publish from the admin does. If the server cannot reach itself at 127.0.0.1:$PORT, it runs them directly and logs a warning; visitors then see what was published only once the cached pages are refreshed.

The timer is off while building, under test, and with JOBS_AUTORUN=false.

Several containers #

Let one container run the jobs and set JOBS_AUTORUN=false on the others. Payload takes a due job in two steps (it finds it, then marks it as taken), so two containers that look at the same moment can both run it, and every container looks at the start of each minute.

Running jobs from outside #

On a platform without a long-running process, or with a scheduler of your own, set JOBS_AUTORUN=false and a CRON_SECRET (openssl rand -hex 32), and call the run endpoint once a minute, from one place only:

curl -fsS -H "Authorization: Bearer $CRON_SECRET" "https://www.example.com/api/payload-jobs/run?allQueues=true"

Vercel Cron sends CRON_SECRET this way by itself. Without the secret the endpoint answers 401; logged-in admins may call it too.

Evaluations, Proofread the site and file translations run for minutes. On a serverless platform, give the function behind the run endpoint a timeout that long. File translation already does one language per run and queues the next.

In the admin #

Admins see the queue under System → Background jobs. Finished jobs are deleted; failed ones stay, with their error. Editors and translators never see it.

A scheduled publish runs with the rights of the person who scheduled it, as they are when it runs. Only people who may publish can schedule.

Adding a job #

Add tasks and workflows to the jobs config in src/jobs/index.ts. A job must also work outside a request, because the timer runs it directly when the server cannot reach itself. Work that must never run twice claims its record with one conditional write (payload.db.updateOne with a where), as file translation does in src/endpoints/translate/fileJob.ts.