Scheduling Tasks

Recently, Caktus has been using Tailscale to manage VPN connections between Android tablets and a central server. We wanted to report on the devices connected to the network using the Tailscale API. While we could use tools like Celery to fetch data from the API and load it into a database—given its widespread use in the Django ecosystem—we also wanted to explore other options.

There are multiple ways to schedule tasks in your Django app, but there are some advantages to using Celery. It’s supported, scales well, and works nicely with Django. Given its wide use, there are also lots of resources for learning more about it, and once learned, that knowledge is likely to be useful on other projects.

When running services in a container, changes to files can be discarded at any time, but the Celery beat default scheduler keeps its state in a file.

(Thanks to Mark Lavin for significant contributions to this post.)

In a previous post, we introduced using Celery to schedule tasks.

In this post, we address things you might need to consider when planning how to deploy Celery in production.

UPDATE: A newer version of this blog post is available for Celery Version 5.0.x.

Getting Started Using Celery for Scheduling Tasks

Many Django applications can make good use of being able to schedule work, either periodically or just not blocking the request thread.

Celery is a great tool for background task processing in Django. We use it in a lot of the custom web apps we build at Caktus, and it's quickly becoming the standard for all variety of task scheduling work loads, from simple to highly complex.