Celery

Recently, Caktus has been using Tailscale to manage VPN connections between Android tablets and a central server. We wanted to report on the devices connected to the network using the Tailscale API. While we could use tools like Celery to fetch data from the API and load it into a database—given its widespread use in the Django ecosystem—we also wanted to explore other options.

There are multiple ways to schedule tasks in your Django app, but there are some advantages to using Celery. It’s supported, scales well, and works nicely with Django. Given its wide use, there are also lots of resources for learning more about it, and once learned, that knowledge is likely to be useful on other projects.

When running services in a container, changes to files can be discarded at any time, but the Celery beat default scheduler keeps its state in a file.

The Celery library (previous posts) makes it as easy to schedule a task to run later as calling a function. Just change:

The weekly Python e-mail newsletter, Pycoder’s Weekly included Caktus Developer Dan Poirier’s post, Celery in Production on their list of the best articles and projects of 2014! Take a look at their list and be sure to revisit Dan’s post.

(Thanks to Mark Lavin for significant contributions to this post.)

In a previous post, we introduced using Celery to schedule tasks.

In this post, we address things you might need to consider when planning how to deploy Celery in production.

Caleb Smith recently gave a talk, “Intro to Celery,” at PyOhio (video below). Celery is a pretty popular topic for us here at Caktus. We use it often in our client work and find it very handy. So we were happy Caleb was out in the world, promoting its use. We sat down with him to hear more about PyOhio and Celery.

UPDATE: A newer version of this blog post is available for Celery Version 5.0.x.

Getting Started Using Celery for Scheduling Tasks

Many Django applications can make good use of being able to schedule work, either periodically or just not blocking the request thread.

Celery is a great tool for background task processing in Django. We use it in a lot of the custom web apps we build at Caktus, and it's quickly becoming the standard for all variety of task scheduling work loads, from simple to highly complex.

Amazon's Simple Queue Service (SQS) is a relatively new offering in the family of Amazon Web Services (AWS). It's also an appealing one, because it proposes to quickly and easily replace a common component of the stack in a typical web application, thereby obviating the need to run a separate queue server like RabbitMQ. While RabbitMQ — the typical favorite for Celery users — is not necessarily difficult to install or maintain, removing it from the stack of a web application means one less component that might fail, offloading that service to AWS — especially for applications with a small to moderate queue volume — might prove financially advantageous.