Tobias McNulty

As a research project this year at Caktus, we decided to build our own private cloud. Some of Caktus' clients prefer or require that their data and workloads remain on premises, which begs the question, "How can we bring a modern, cloud-like experience to an on-premises data center?"

One of my favorite parts of working at Caktus is the community of developers who share knowledge within and outside our teams. Many of us work on different projects, but still take time to come together, share what we've learned, and apply those learnings to other projects.

Nearly every week, I receive an email or two from a third party expressing interest in buying Caktus. As a matter of habit, I don't open them, let alone respond. Most are scattershot, venture capital firms looking for Software-as-a-Service companies (which Caktus is not). But when an employee approached me in 2018 expressing an interest in making Caktus employee-owned, I listened.

We regularly update and improve our upgrades and sustainability service offerings at Caktus, and a recent upgrade for a client precipitated a solution that I felt might be worth sharing. At Caktus, the preferred approach for addressing upgrades and sustainability is to make incremental updates to a project over time, trying to keep both Django and the servers themselves on a long-term support version. These are select versions of Django and Ubuntu, for example, that generally have much longer support periods than other versions, i.e., they are a good fit for applications that you will need to continue maintaining well into the future.

There exists on Docker hub a Docker image called docker. It also has two flavors, "stable" and a "dind" (Docker-in-Docker). What is this image for and what is the purpose of these two different image tags?

For many years, we've been running an ELK (Elasticsearch, Logstash, Kibana) stack for centralized logging. We have a specific project that requires on-premise infrastructure, so sending logs off-site to a hosted solution was not an option. Over time, however, the maintenance requirements of this self-maintained ELK stack were staggering. Filebeat, for example, filled up all the disks on all the servers in a matter of hours, not once, but twice (and for different reasons) when it could not reach its Logstash/Elasticsearch endpoint. Metricbeat suffered from a similar issue: It used far too much disk space relative to the value provided in its Elasticsearch indices. And while provisioning a self-hosted ELK stack has gotten easier over the years, it's still a lengthy process, which requires extra care anytime an upgrade is needed. Are these problems solvable? Yes. But for our needs, a simpler solution was needed.

The Django documentation recommends always starting your project with a custom user model (even if it's identical to Django's to begin with), to make it easier to customize later if you need to. But what are you supposed to do if you didn't see this when starting a project, or if you inherited a project without a custom user model and you need to add one?

It's been awhile since we last discussed bulk inserts on the Caktus blog. The idea is simple: if you have an application that needs to insert a lot of data into a Django model — for example a background task that processes a CSV file (or some other text file) — it pays to "chunk" those updates to the database so that multiple records are created through a single database operation. This reduces the total number of round-trips to the database, something my colleague Dan Poirier discussed in more detail in the post linked above.

Dokku is a great little tool. It lets you set up your own virtual machine (VM) to facilitate quick and easy Heroku-like deployments through a git push command. Builds are fast, and updating environment variables is easy. The problem is that Dokku includes all of your services on a single instance. When you run your database on the Dokku instance, you risk losing it (and any data that's not yet backed up) should your VM suddenly fail.

We deploy Python/Django apps to a wide variety of hosting providers at Caktus. Our django-project-template includes a Salt configuration to set up an Ubuntu virtual machine on just about any hosting provider, from scratch. We've also modified this a number of times for local hosting requirements when our customer required the application we built to be hosted on hardware they control. In the past, we also built our own tool for creating and managing EC2 instances automatically via the Amazon Web Services (AWS) APIs. In March, my colleague Dan Poirier wrote an excellent post about deploying Django applications to Elastic Beanstalk demonstrating how we’ve used that service.