How-To on Caktus Group

Analyze data with SQL window functions

Wed, 05 Apr 2023 14:00:00 +0000

We regularly use tools like PostgreSQL, Pandas, and Jupyter Notebooks to analyze data here at Caktus. Recently, we were reviewing North Carolina traffic stop data for the NC CopWatch project and had the opportunity to use PostgreSQL's window functions, which are helpful when aggregating data.

How to Use Celery for Scheduling Tasks

Wed, 11 Aug 2021 18:14:40 +0000

There are multiple ways to schedule tasks in your Django app, but there are some advantages to using Celery. It’s supported, scales well, and works nicely with Django. Given its wide use, there are also lots of resources for learning more about it, and once learned, that knowledge is likely to be useful on other projects.

How Mock Can Improve Your Unit Tests: Part 2

Mon, 05 Apr 2021 12:54:21 +0000

In Part 1 of this blog series, we started looking at how we can use mocking to improve our unit tests. We'll expand on that in this post.

How Mock Can Improve Your Unit Tests: Part 1

Mon, 29 Mar 2021 12:54:20 +0000

We've written in the past about the importance of unit tests and how to write better ones: Culture of Unit Testing, Subtests are the Best.

Automating PyPI releases with Github Actions

Thu, 11 Feb 2021 14:37:42 +0000

At Caktus, we maintain some open source packages, which are usually pieces of code that we’ve found useful when building an application, and we figure they might be useful to others. From time to time, those open source projects need a little maintenance: Maybe a bug fix has been provided from a community member, or we need to update to support a new version of Python or Django. I always hesitate to do this because I know there are multiple steps involved to ensure that any updates are properly tested, documented, versioned, and released. One of my silly blockers is that I always seem to forget how to upload a release to PyPI, which is the Python Package Index, a repository of software for Python. I call it silly because it shouldn’t really be a blocker, seeing how simple it actually is, but I still forget it each time and have to walk through the documentation step by step to make sure I do it properly. I recently found a tool that helps with this process.

JavaScript Variable Declaration in the 21st Century

Mon, 25 Jan 2021 15:32:15 +0000

October 2020 marked the 14th anniversary of the introduction of “let” and "const" to Javascript. Yes, it was in 2006 that this feature was first introduced, and we’re still having conversations about it. So why, you ask, are we still talking about this? It seems that a lot of confusion remains, not to mention a lot of old code being passed around on StackOverflow with “var” all over the place. Here I’ll discuss the difference between these three declarations, and come to a conclusion about best practices

My Python Development Environment

Thu, 14 Jan 2021 14:40:35 +0000

This is how to set up the One True Development Environment for Python.

Haha, just kidding, there is no such thing. Here's one way to do it that works for me, and an attempt to explain the benefits of doing it this way.

Our Top 20 Blogs in 2020

Tue, 05 Jan 2021 15:26:55 +0000

We’ve published summaries of our most popular blog posts before (see Top 19 of 2019 and Top 18 of 2018), but this time, we’re taking it a step further. We’re sharing the 20 most popular posts in 2020, regardless of the year the post was originally published. And some of these have been around a while! Based on total pageviews, here are the blogs that rose to the top of the popularity list, from most viewed to least viewed.

Build a useKeypress Hook in React

Wed, 01 Jul 2020 19:27:20 +0000

One of the great patterns to come out of React 16.8 is composable hooks. Using React's built-in hooks such as useState and useEffect, we can encapsulate and modularize bits of functionality — almost the same way we create reusable components. In this article we're going to throw together a quick hook that registers an event listener on a keypress and performs an action. Nothing fancy, but it's a nice abstraction that makes it a lot easier to add some nice power-user features to your app.

How to Schedule Tasks Using Celery Beat in a Container

Thu, 14 May 2020 12:30:00 +0000

When running services in a container, changes to files can be discarded at any time, but the Celery beat default scheduler keeps its state in a file.

How to Use Kubernetes on Azure for Cloud Computing

Tue, 28 Apr 2020 18:41:03 +0000

For the Review, Appraisal, and Triage of Mail (RATOM) project, funded by the Andrew W. Mellon Foundation, we were tasked with deploying to a Microsoft Azure environment. More details about the project are in our first blog post in this Learn With Us blog series. Caktus has experience with Amazon Web Services (AWS) and Google Cloud, but we hadn't had the opportunity to use Azure yet, so we looked forward to the opportunity to use that environment and document our experience. The entire deployment process is available on GitHub as a reference under the StateArchivesOfNorthCarolina/ratom-deploy repository.

What to do About Email: How to Extract Data from Microsoft PST Files

Thu, 19 Mar 2020 12:27:52 +0000

In my previous line of work as an archivist, the question of what to do about email archives was an ongoing and deeply-considered topic. Email is everywhere. Yes, even Gen Z and millennials use it, despite thousands of think pieces that would have you believe that the old ways are giving way to business meetings conducted on fixed-gear bicycles, over avocado toast and Instagram.

Creating a Sub-account in Amazon Web Services

Tue, 03 Mar 2020 20:20:50 +0000

Anytime we host resources for a client within the Caktus Amazon Web Services (AWS) account, we set up a sub-account and put the resources there. Some of the advantages of doing this compared to putting a client's resources in our main account are:

How to Use the "docker" Docker Image to Run Your Own Docker daemon

Tue, 25 Feb 2020 13:00:00 +0000

There exists on Docker hub a Docker image called docker. It also has two flavors, "stable" and a "dind" (Docker-in-Docker). What is this image for and what is the purpose of these two different image tags?

Our Top 19 Blogs of 2019

Wed, 01 Jan 2020 17:17:32 +0000

During the last year we gave our popular technical blog an official name: Developer Access. We published 32 posts on the blog, including technical how-to’s, conference information, web development best practices and detailed guides. Among all those posts, 19 rose to the top of the popularity list (based on total pageviews):

How to Do Wagtail Data Migrations

Thu, 12 Sep 2019 20:12:03 +0000

Wagtail is a fantastic content management system that does a great job of making it easy for developers to get a new website up and running quickly and painlessly. It’s no wonder that Wagtail has grown to become the leading Django-based CMS. As one of the creators of Wagtail recently said, it makes the initial experience of getting a website set up and running very good. At Caktus, Wagtail is our go-to framework when we need a content management system.

How to Import Multiple Excel Sheets in Pandas

Tue, 13 Aug 2019 12:00:00 +0000

Pandas is a powerful Python data analysis tool. It's used heavily in the data science community since its data structures make real-world data analysis significantly easier. At Caktus, in addition to using it for data exploration, we also incorporate it into Extract, Transform, and Load (ETL) processes.

How to Set Up a Centralized Log Server with rsyslog

Thu, 20 Jun 2019 12:00:00 +0000

For many years, we've been running an ELK (Elasticsearch, Logstash, Kibana) stack for centralized logging. We have a specific project that requires on-premise infrastructure, so sending logs off-site to a hosted solution was not an option. Over time, however, the maintenance requirements of this self-maintained ELK stack were staggering. Filebeat, for example, filled up all the disks on all the servers in a matter of hours, not once, but twice (and for different reasons) when it could not reach its Logstash/Elasticsearch endpoint. Metricbeat suffered from a similar issue: It used far too much disk space relative to the value provided in its Elasticsearch indices. And while provisioning a self-hosted ELK stack has gotten easier over the years, it's still a lengthy process, which requires extra care anytime an upgrade is needed. Are these problems solvable? Yes. But for our needs, a simpler solution was needed.

How to Switch to a Custom Django User Model Mid-Project

Fri, 26 Apr 2019 16:32:33 +0000

The Django documentation recommends always starting your project with a custom user model (even if it's identical to Django's to begin with), to make it easier to customize later if you need to. But what are you supposed to do if you didn't see this when starting a project, or if you inherited a project without a custom user model and you need to add one?

Coding for Time Zones & Daylight Saving Time — Oh, the Horror

Thu, 21 Mar 2019 16:45:36 +0000

In this post, I review some reasons why it's really difficult to program correctly when using times, dates, time zones, and daylight saving time, and then I'll give some advice for working with them in Python and Django. Also, I'll go over why I hate daylight saving time (DST).

Suggestions For Picking Up Old Projects

Fri, 15 Feb 2019 18:36:49 +0000

At Caktus, we work on many projects, some of which are built by us from start to finish, while others are inherited from other sources. Oftentimes, we pick up a project that we either have not worked on in a long time, or haven’t worked on at all, so we have to get familiar with the code and figure out the design decisions that were made by those who developed it (including when the developers are younger versions of ourselves). Moreover, it is a good idea to improve the setup process in each project, so others can have an easier time getting set up in the future. In our efforts to work on such projects, a few things have been helpful both for becoming familiar with the projects more quickly, and for making the same projects easier to pick up in the future.

A Guide To Creating An API Endpoint With Django Rest Framework

Fri, 01 Feb 2019 20:42:28 +0000

As part of our work to make sharp web apps at Caktus, we frequently create API endpoints that allow other software to interact with a server. Oftentimes this means using a frontend app (React, Vue, or Angular), though it could also mean connecting some other piece of software to interact with a server. A lot of our API endpoints, across projects, end up functioning in similar ways, so we have become efficient at writing them, and this blog post gives an example of how to do so.

How to Use Django Bulk Inserts for Greater Efficiency

Wed, 09 Jan 2019 13:00:00 +0000

It's been awhile since we last discussed bulk inserts on the Caktus blog. The idea is simple: if you have an application that needs to insert a lot of data into a Django model — for example a background task that processes a CSV file (or some other text file) — it pays to "chunk" those updates to the database so that multiple records are created through a single database operation. This reduces the total number of round-trips to the database, something my colleague Dan Poirier discussed in more detail in the post linked above.

Caktus Blog: Top 18 Posts of 2018

Tue, 01 Jan 2019 13:00:00 +0000

In 2018, we published 44 posts on our blog, including technical how-to’s, a series on UX research methods, web development best practices, and tips for project management. Among all those posts, 18 rose to the top of the popularity list in 2018.

How to Fix your Python Code's Style

Mon, 10 Dec 2018 12:00:00 +0000

Sometimes we inherit code that doesn't follow the style guidelines we prefer when we're writing new code. We could just run flake8 on the whole codebase and fix everything before we continue, but that's not necessarily the best use of our time.

Filtering and Pagination with Django

Thu, 18 Oct 2018 14:40:42 +0000

If you want to build a list page that allows filtering and pagination, you have to get a few separate things to work together. Django provides some tools for pagination, but the documentation doesn't tell us how to make that work with anything else. Similarly, django_filter makes it relatively easy to add filters to a view, but doesn't tell you how to add pagination (or other things) without breaking the filtering.

Better Python Dependency Management with pip-tools

Tue, 18 Sep 2018 14:37:29 +0000

I recently looked into whether I could use pip-tools to improve my workflow around projects' Python dependencies. My conclusion was that pip-tools would help on some projects, but it wouldn't do everything I wanted, and I couldn't use it everywhere. (I tried pip-tools version 2.0.2 in August 2018. If there are newer versions, they might fix some of the things I ran into when trying pip-tools.)

Make ALL Your Django Forms Better

Mon, 18 Jun 2018 13:00:00 +0000

Website experiences need to be consistent as much as they need to be well thought out and aesthetically pleasing. Structure, visual design, user interactions, and accessibility concerns are among many considerations that go into building quality websites. While achieving consistency of experience and implementation is an essential goal of web development, efficiency of execution is also very important. An efficient workflow means this consistent experience doesn’t require redoing work across the site.

Creating Dynamic Forms with Django

Mon, 07 May 2018 13:30:00 +0000

Editor’s note: This post was written in 2018 when Django 1.11 and Django 2.0 were current. The code samples may need to be updated to work with more recent versions of Django.

Supercharging your CSS with Stylus and PostCSS

Mon, 18 Dec 2017 14:30:00 +0000

Here at Caktus the front-end team stays on the bleeding edge by taking advantage of the latest and greatest tools. We only incorporate features into our packaging that are well-supported and production-ready, as well as those that meet our list of standard browser requirements. Luckily, there are plenty of tools that allow us to use experimental technologies with appropriate fallbacks for non-supported browsers.

Building a Custom Block Template Tag

Mon, 01 May 2017 13:30:00 +0000

Building custom tags for Django templates has gotten much easier over the years, with decorators provided that do most of the work when building common, simple kinds of tags.

Learning to ask the right questions, or people

Mon, 10 Apr 2017 13:30:00 +0000

I ask a lot of questions as a developer. Some of them have been more basic, like ‘How do I import a Python function from one file into another?’, and some more complex, like ‘How should we take an API request and return a dynamically-generated PDF as a response?’

How to write a bug report

Wed, 15 Feb 2017 13:00:00 +0000

Here are some brief thoughts on writing good bug reports in general.

Main elements

There are four crucial elements when writing a bug report:

New white paper: "Shipping faster: Django team improvements"

Mon, 04 Apr 2016 13:26:38 +0000

For the past couple months, we’ve been working on a new white paper, “Shipping Faster: Django Team Improvements”. We examined our existing processes, looked at best practices, and considered what has or hasn’t worked across our dozens of simultaneous projects.

AWS load balancers with Django

Mon, 10 Aug 2015 12:00:00 +0000

We recently had occasion to reconfigure some of our existing servers to use Amazon Web Services Elastic Load Balancers in front of them. Setting this up isn’t hard, exactly, but there are a lot of moving parts that have to mesh correctly before things start to work, so I thought I’d write down what we did.

Django Logging Configuration: How the Default Settings Interfere with Yours

Tue, 27 Jan 2015 16:04:04 +0000

My colleague Vinod recently found the answer on Stack Overflow to something that’s been bugging me for a long time - why do my Django logging configurations so often not do what I think they should?

Webinar: Testing Client-Side Applications with Django

Tue, 20 Jan 2015 16:40:09 +0000

Technical Director Mark Lavin will be hosting a free O’Reilly webinar today at 4PM EST or 1PM PT on Testing Client-Side Applications with Django. Mark says testing is one of the most popular question topics he receives. It’s also a topic near and dear to Caktus’ quality-loving heart. Mark’s last webinar garnered more than 500 viewers, so sign up quick!

Using Amazon S3 to Store your Django Site's Static and Media Files

Mon, 10 Nov 2014 16:28:11 +0000

Editor's note: This post was updated in September 2017.

Using Amazon S3 to Store your Django Site's Static and Media Files

Storing your Django site's static and media files on Amazon S3, instead of serving them yourself, can improve site performance. It frees your servers from handling static files themselves, lets you scale your servers easier by keeping media files in a common place, and is a necessary step to using Amazon CloudFront as a Content Delivery Network (CDN).

Webcast: Creating Enriching Web Applications with Django and Backbone.js

Tue, 04 Nov 2014 15:00:00 +0000

Update: The live webcast is now available at O’Reilly Media

Our technical director, Mark Lavin, will be giving a tutorial on Django and Backbone.js during a free webcast for O’Reilly Media tomorrow, November 6th, 1pm EST. There will be demos and a discussion of common stumbling blocks when building rich client apps.

Celery in Production

Mon, 29 Sep 2014 12:00:00 +0000

(Thanks to Mark Lavin for significant contributions to this post.)

In a previous post, we introduced using Celery to schedule tasks.

In this post, we address things you might need to consider when planning how to deploy Celery in production.

Getting Started Scheduling Tasks with Celery

Mon, 23 Jun 2014 11:00:00 +0000

UPDATE: A newer version of this blog post is available for Celery Version 5.0.x.

Getting Started Using Celery for Scheduling Tasks

Many Django applications can make good use of being able to schedule work, either periodically or just not blocking the request thread.

Managing Events with Explicit Time Zones

Thu, 09 Jan 2014 14:00:00 +0000

Managing Events with Explicit Time Zones

Recently we wanted a way to let users create real-life events which could occur in any time zone that the user desired. By default, Django interprets any date/time that the user enters as being in the user’s time zone, but it never displays that time zone, and it converts the time zone to UTC before storing it, so there is no record of what time zone the user initially chose. This is fine for most purposes, but if you want to specifically give the user the ability to choose different time zones for different events, this won’t work.

Using strace to Debug Stuck Celery Tasks

Wed, 30 Oct 2013 19:56:31 +0000

Celery is a great tool for background task processing in Django. We use it in a lot of the custom web apps we build at Caktus, and it's quickly becoming the standard for all variety of task scheduling work loads, from simple to highly complex.

Skipping Test DB Creation

Wed, 02 Oct 2013 12:48:59 +0000

We are always looking for ways to make our tests run faster. That means writing tests which don't preform I/O (DB reads/writes, disk reads/writes) when possible. Django has a collection of TestCase subclasses for different use cases. The common TestCase handles the fixture loading and the creation the of TestClient. It uses the database transactions to ensure that the database state is reset for every test. That is it wraps each test in a transaction and rolls it back once the test is over. Any transaction management inside the test becomes a no-op. Since [TestCase]{.title-ref}[ overrides the transaction facilities, if you need to test the transactional behavior of a piece of code you can instead use ]{.title-ref}[TransactionTestCase]{.title-ref}[. ]{.title-ref}[TransactionTestCase]{.title-ref}` resets the database after the test runs by truncating all tables which is much slower than rolling back the transaction particularly if you have a large number of tables.

Central logging in Django with Graylog2 and graypy

Wed, 18 Sep 2013 14:00:12 +0000

Django's logging configuration facilities, which arrived in version 1.3, have greatly eased (and standardized) the process of configuring logging for Django projects. When building complex and interactive web applications at Caktus, we've found that detailed (and properly configured!) logs are key to successful and efficient debugging. Another step in that process — which can be particularly useful in environments where you have multiple web servers — is setting up a centralized logging server to receive all your logs and make them available through an easily accessible web interface. There are a number useful tools to do this, but one we've found that works quite well is Graylog2. Installing and configuring Graylog2 is outside the scope of this post, but there are plenty of tutorials on how to do so accessible through your search engine of choice.

Migrating to a Custom User Model in Django

Wed, 07 Aug 2013 13:10:00 +0000

UPDATE: Read a newer blog post on this topic.

The new custom user model configuration that arrived in Django makes it relatively straightforward to swap in your own model for the Django user model. In most cases, Django's built-in User model works just fine, but there are times when certain limitations (such as the length of the email field) require a custom user model to be installed. If you're starting out with a custom user model, setup and configuration are relatively straightforward, but if you need to migrate an existing legacy project (e.g., one that started out in Django 1.4 or earlier), there are a few gotchas that you might run into. We did this recently for one of our larger, long-term client projects at Caktus, and here's an outline of how we'd recommend tackling this issue:

MEDIA_ROOT and Django Tests

Wed, 26 Jun 2013 14:54:30 +0000

If you’ve ever written a test for a view or model with associated uploaded files you might have noticed a small problem with those files hanging around after the tests are complete. Since version 1.3, Django won’t delete the files associated with your model instances when they are deleted. Some work-arounds for this issue involve writing a custom delete for your model or using a post_delete signal handler. But even with those in place the files would not be deleted during tests because the model instances are not explicitly deleted at the end of the test case. Instead, Django simply rolls back the transaction and the delete method is never called nor are the signals fired. This can be quite an annoyance when running the tests repeatedly and watching your MEDIA_ROOT (or worse your S3 bucket) fill up with garbage data. More than annoyance, this introduces something you always want to avoid in unittests: global state.

Making your Django app more pluggable

Wed, 12 Jun 2013 12:08:27 +0000

This blog post is about ways of doing things in your Django app, and pitfalls to avoid, so your app is easier to plug in to other sites.

Managing Django Translations with Transifex

Thu, 25 Apr 2013 15:51:27 +0000

We manage a number of open source Django apps here at Caktus. While many of us here are proficient in a number of programming languages, the same can’t be said for our ability to read or write in languages other than English. The Django community is global and we want our apps to support other languages. For that we’ve turned to Transifex which it the same tool Django itself uses for translations.

A helper script for runserver

Thu, 11 Apr 2013 16:17:37 +0000

My Django runserver shortcut script

This is a little shell script I save as rs, put on my path, and use as a shortcut to start the Django development server. It has several features that make me more productive:

Using LESS with Django

Mon, 05 Mar 2012 14:00:00 +0000

Lately, I’ve been working on creating a simplified work flow for my front end work here at Caktus. There are all sorts of new and helpful tools to optimize the creative process, allowing for faster iterations, and greater overall enjoyment. As with any new tool, there are a few options to choose from: LESS and SASS. Having read lots of reviews and reading through the documentation, I’ve decided LESS is more for me.

Configuring a Jenkins Slave

Tue, 10 Jan 2012 13:50:00 +0000

We're pretty avid testers here at Caktus and when one of our Django projects required upgrading to Python 2.7, we also needed to upgrade our Jenkins build environment. Luckily, Jenkins supports distributed builds to allow a master install to delegate tasks to slaves instances. This way we can continue to run our primary build system on Ubuntu 10.04, which defaults to Python 2.6, and delegate tasks to an Ubuntu 11.04 environment running Python 2.7. The setup is fairly easy, but since I didn't find much out there already, I figured I write up a quick post outlining what we did.

Class-based views in Django 1.3

Thu, 29 Dec 2011 11:00:00 +0000

Django class-based views

Introduction

Django 1.3 added class-based views, but neglected to provide documentation to explain what they were or how to use them. So here's a basic introduction.

OpenBlock Geocoder, Part 3: External Geocoders

Wed, 28 Dec 2011 13:23:21 +0000

The OpenBlock geocoder is powerful and robust. It uses PostGIS for spacial queries, can extract addresses from bodies of text, and can understand block and intersection notation. We've run into a few issues with it, however, including a low geocoding success rate. This is a tough problem to solve and depends on a lot of factors (the extent of street and block data in OpenBlock, format of the street addresses, etc.), so your mileage may vary. Below I constructed a simple test using Google's Geocoding API to have as an alternative.

Using Django and Celery with Amazon SQS

Mon, 19 Dec 2011 16:26:48 +0000

Amazon's Simple Queue Service (SQS) is a relatively new offering in the family of Amazon Web Services (AWS). It's also an appealing one, because it proposes to quickly and easily replace a common component of the stack in a typical web application, thereby obviating the need to run a separate queue server like RabbitMQ. While RabbitMQ — the typical favorite for Celery users — is not necessarily difficult to install or maintain, removing it from the stack of a web application means one less component that might fail, offloading that service to AWS — especially for applications with a small to moderate queue volume — might prove financially advantageous.

OpenBlock Geocoder, Part 2: Text Parsing and Entity Extraction

Fri, 16 Dec 2011 13:30:19 +0000

This is the second post in our OpenRural series reviewing OpenBlock and it's geocoder. OpenBlock Geocoder, Part 1: Data Model and Geocoding covers the internals of the OpenBlock geocoder and it's geocoding capabilities. As this posts builds upon topics covered there, you may wish to read Part 1 before proceeding. In this post we step back from the internals of the geocoder and explore how to use it along with other OpenBlock tools to parse unstructured text.

OpenBlock Geocoder, Part 1: Data Model and Geocoding

Mon, 12 Dec 2011 16:17:20 +0000

As Tobias mentioned in Scraping Data and Web Standards, Caktus is collaborating with the UNC School of Journalism to help develop Open Rural (the code is on GitHub). Open Rural hopes to help rural newspapers in North Carolina leverage OpenBlock. This blog post is the first of several covering the internals of OpenBlock and, specifically, the geocoder.

Django Without the Web

Mon, 24 Oct 2011 16:00:00 +0000

One of the things I like best about Django is how easy its ORM makes it to work with databases. Too bad Django is only for web applications. Sure, you could deploy a Django app and then make use of it from a non-web application using a REST API, but that would be too awkward.

Bulk inserts in Django

Tue, 20 Sep 2011 18:08:52 +0000

I recently found a way to speed up a large data import far more than I expected.

The task was to read data from a text file and create data records in Django, and the naive implementation was managing to import about 55 records per second, which was going to take far too long given the amount of data that needed to be imported.

Testing Web Server Configurations with Fabric and ApacheBench

Tue, 13 Sep 2011 18:16:33 +0000

Load testing a site with ApacheBench is fairly straight forward. Typically you'd just SSH to a machine on the same network as the one you want to test, and run a command like this:

Getting Started using Python in Eclipse

Wed, 31 Aug 2011 19:05:53 +0000

Eclipse with the PyDev module has a lot to offer the Python programmer these days. If you haven't looked at PyDev before, or not in a while, it's worth checking out.

Simplifying the Testing of Unmanaged Database Models in Django (Updated for Django 4.2 in 2024)

Fri, 24 Sep 2010 15:01:09 +0000

Editor's note: This post was originally published in September, 2010 and was updated in December, 2024 to incorporate changes in Django and improvements suggested by our readers. It has also been tested for compatibility as of the Django 4.2 release.

Basic Django deployment with virtualenv, fabric, pip and rsync

Fri, 23 Apr 2010 00:46:37 +0000

Deployment is usually a tedious process with lots of tinkering until everything is setup just right. We deploy quite a few Django sites on a regular basis here at Caktus and still do tinkering, but we've attempted to functionalize some of the core tasks to ease the process. I've put together a basic example that outlines local and remote environment setup. This is a simplified example and just one of many ways to deploy a Django project (I learned a lot from Jacob Kaplan-Moss' django-deployment-workshop), so I encourage you to browse around the Django community to learn more. The entire source for this example project can be found in the caktus-deployment Bitbucket repository.

Decoupled Django Apps and the Beauty of Generic Relations

Thu, 11 Mar 2010 21:18:02 +0000

Like just about everyone else, we've written our own suite of tools to help with building complex content management systems in Django here at Caktus. We reviewed a number of the existing CMSes out there, but in almost every case the navigation and page structure were so tightly coupled the system broke down when it came time to add additional, non-CMS pages.

Continuous Integration with Django and Hudson CI (Day 1)

Mon, 08 Mar 2010 15:58:48 +0000

We're always looking for new tools to make our development environment more robust here at Caktus. We write a lot of tests to ensure proper functionality as new features land and bug fixes are added to our projects. The next step is to integrate with a continuous integration system to automate the process and regularly check that status of the build.

Custom JOINs with Django's query.join()

Mon, 28 Sep 2009 14:26:02 +0000

Django's ORM is great. It handles simple to fairly complex queries right out the box without having to write any SQL. If you need a complicated query, Django's lets you use .extra(), and you can always fallback to raw SQL if need be, but then you lose the ORM's bells and whistles. So it's always nice to find solutions that allow you to tap into the ORM at different levels.

Creating recursive, symmetrical many-to-many relationships in Django

Fri, 14 Aug 2009 18:10:43 +0000

In Django, a recursive many-to-many relationship is a ManyToManyField that points to the same model in which it's defined ('self'). A symmetrical relationship is one in where, when a.contacts = [b], a is in b.contacts.

Setting PostgreSQL's SHMMAX in Mac OS X 10.5 (Leopard)

Thu, 13 Aug 2009 11:46:08 +0000

If you've ever tried to increase the shared_buffers setting in your postgresql.conf to a value that exceeds the amount of shared memory supported by your operating system kernel, then you'll see an error message like this:

Towards a Standard for Django Session Messages

Fri, 19 Jun 2009 15:31:13 +0000

Django needs a standard way in which session-specific messages can be created and retrieved for display to the user. For years we’ve been surviving using user.message_set to store messages that are really specific to the current session, not the user, or using the latest and greatest Django snippet, pluggable app, or custom crafted middleware to handle messages in a more appropriate way.

Remote logging with Python logging and Django

Tue, 09 Jun 2009 12:47:10 +0000

As part of my work on EveryWatt, our fledgling energy monitoring web site, I needed a way to consolidate log messages from all the data loggers we have running in a single place. If you're not familiar with it, Python's logging module is good stuff and worth checking out. We already used it for logging to files locally, and the module defines an HTTPHandler that can deliver log messages to a remote server via HTTP.

Testing Django Views for Concurrency Issues (Updated for Django 4.2 in 2024)

Tue, 26 May 2009 19:59:25 +0000

Editor's note: This post was originally published in May, 2009 and was updated in December, 2024 to incorporate changes in Django and improvements suggested by our readers. It has also been tested for compatibility as of the Django 4.2 release.

Explicit Table Locking with PostgreSQL and Django

Tue, 26 May 2009 18:59:10 +0000

By default, Django doesn't do explicit table locking. This is OK for most read-heavy scenarios, but sometimes you need guaranteed, exclusive access to the data. Caktus uses PostgreSQL in most of our production environments, so we can use the various lock modes it provides to control concurrent access to the data. Once we obtain a lock in PostgreSQL, it is held for the remainder of the current transaction. Django provides transaction management, so all we need to do is execute a SQL LOCK statement within a transaction, and Django and PostgreSQL will handle the rest.

Parsing Microseconds in a Django Form

Tue, 26 May 2009 13:43:56 +0000

There's currently no way to accept microsecond-precision input through a Django form's DateTimeField. This is an acknowledged bug, but the official solution might not come very soon, because the real fix is non-trivial.

Seamlessly switch off (and on) a Django (or other WSGI) site for upgrades

Mon, 25 May 2009 19:53:18 +0000

In preparation for migrating the EveryWatt database from one machine to another, I wrote this little WSGI script to easily disable the site while I copy the data. Since it doesn't depend on Django or really anything else (other than a functioning WSGI server), you can use it for other upgrades, too.

Eclipse Ganymede and Subclipse on Ubuntu - JavaHL (JNI) not available

Thu, 21 May 2009 15:10:25 +0000

I finally got around to updating my Eclipse, PyDev, and Subclipse environment today, which I use for Django development.

Formerly I was using the SvnKit (pure-Java) libraries. SvnKit "felt" slow to me, compared to my command line SVN client, so this time I tried to get the JavaHL (JNI) libraries working.

Downsizing an LVM/RAID root partition

Thu, 23 Apr 2009 20:04:03 +0000

At Caktus we use LVM2 on a RAID1 device to ease disk management on a number of our servers. Recently I needed to downsize the root partition of one of the servers, so I rebooted onto an Ubuntu 8.10 LiveCD and attempted to load the RAID/LVM info.

Migrating from django-photologue 1.x to 2.x

Fri, 27 Mar 2009 17:52:30 +0000

We're in the process of updating a web app for a client that was built last year about this time using Django and Photologue. Needless to say, there have been a lot of changes to both over the past year!

Solving NFS issues on embedded machines

Wed, 25 Mar 2009 16:07:27 +0000

As part of my work on EveryWatt, I setup an NFS-based development environment for one of the data loggers we use for energy monitoring in the Caktus office. The stock 2.4 Linux kernel in the machine seemed to have some trouble mounting the file system root I had exported from one of our servers. The symptoms included long delays for most if not all activities that used the file system and lots of messages like these in dmesg:

Overriding Django admin templates for fun and profit

Tue, 20 Jan 2009 18:25:48 +0000

Motivation & Goal

I sometimes find the admin interface's lists of instances of models overwhelming. The filters and searching helps, but it would be nice to get an overview of the data that is being shown. Particularly I wanted to generate a graph based on the filters selected by the user, so that only items displayed after a filter would be graphed.

Entering Contacts in CiviCRM

Wed, 05 Nov 2008 20:29:51 +0000

One of our non-profit clients recently asked for help entering the numerous business cards they get at trade shows, etc., into their Customer Relationship Management database, a copy of CiviCRM that we setup and manage for them. The best path for entering contacts isn't necessarily obvious from the get-go, but the following procedure is the best we've found and has the lowest up-front investment (you might be able to do something more efficient with a Profile, but that doesn't seem as flexible with respect to matching existing contacts).

Is Asterisk More Silent Than You'd Like?

Sat, 15 Dec 2007 19:06:01 +0000

At Caktus, we use the Asterisk open source telephony platform. While setting up Asterisk, we tested a number of different system environments and configurations to help determine what would best fit our telephone needs. This involved running Asterisk on both real and virtual machines and testing several configuration front ends, including FreePBX and DeStar. This article discusses a fairly technical problem we confronted while installing a basic Asterisk setup on the Debian GNU/Linux operating system (on real hardware); namely, when the ztdummy Linux kernel module was loaded, Asterisk was unable to play back any recordings (even its own). When we unloaded the ztdummy module, however, recordings would play back fine. The catch, of course, was that we needed ztdummy loaded for conference room support.