Django is a tool we use every day to build fantastic web apps here at Caktus, and a development sprint is a concerted, focused period of time in which developers meet in the same space to get things done on a project.

We’re proud to annouce that Caktus is hosting another local Django development sprint in the Triangle (Raleigh, Durham, and Chapel Hill/Carrboro) area of North Carolina. The sprint will be held the weekend of March 20th and 21st in Carrboro Creative Coworking, and the purpose of this sprint will be to help push out bug fixes in preparation for the upcoming Django 1.2 release.

If you’re interested in attending, no previous experience contributing to Django is necessary and the sprint will be a great opportunity to start. Work on other open source Django-based projects is welcome too. For more information, check out the corresponding wiki page.

We’ll be there to open the doors at 9am both days. Courtesy of our sponsors there will be free drinks, snacks, and lunch to go around. Hope to see you there!

Like just about everyone else, we’ve written our own suite of tools to help with building complex content management systems in Django here at Caktus. We reviewed a number of the existing CMSes out there, but in almost every case the navigation and page structure were so tightly coupled the system broke down when it came time to add additional, non-CMS pages.

We wrote a few little apps, django-pagelets, django-treenav, and django-crumbs, each of which manages different pieces of content (little snippets of content, full CMS pages, navigation, and breadcrumbs). All of the apps are available for free under an open source license on Google Code.

Decoupling was a great move for us, and the ability to plug and play any single part of the system is a huge benefit. Sometimes, however, the completely decoupled architecture was a bit of a pain: If we didn’t provide a link from the pagelets app to the treenav app, how would it be possible to edit a page’s corresponding navigation item on its change form in the Django admin interface?

Enter Generic Relations. Using Django’s content types framework, it’s possible to create admin inlines for generic relations with just a few simple lines of code.

In this case, I’ll show how we allowed users to edit a page’s corresponding navigation item in django-pagelets without requiring everyone (i.e., those who don’t need it) to install django-treenav. First, define the generic inline in the admin.py file of the app that contains the model you want to link to:

from django.contrib.contenttypes import generic
class GenericMenuItemInline(generic.GenericStackedInline):
    """
    Add this inline to your admin class to support editing related menu items
    from that model's admin page.
    """
    max_num = 1
    model = treenav.MenuItem

Then, inside the Admin class for the related model in question, dynamically import and add GenericMenuItemInline to the admin’s list of inlines based on whether or not it’s in the project’s INSTALLED_APPS:

from django.conf import settings
class PageAdmin(admin.ModelAdmin):
    # ...
    inlines = [MyOtherInline]
    if 'treenav' in settings.INSTALLED_APPS:
        from treenav.admin import GenericMenuItemInline
        inlines.insert(0, GenericMenuItemInline)

For more information, see the corresponding pagelets admin.py and treenav admin.py. Thanks for reading and don’t hesitate to post comments if you have any questions!

We’re always looking for new tools to make our development environment more robust here at Caktus. We write a lot of tests to ensure proper functionality as new features land and bug fixes are added to our projects. The next step is to integrate with a continuous integration system to automate the process and regularly check that status of the build.

After attending Dr. C. Titus Brown’s “Why not run all your tests all the time? A study of continuous integration systems.” talk at Pycon and seeing Django’s Hudson setup, I figured I’d take a look at Hudson CI.

Installing Hudson and basic setup

Hudson is very easy to setup. I started with a fresh Ubuntu 9.10 install on the smallest Rackspace cloud instance and had it running after a few commands. I followed the Debian setup instructions, which basically consists of:

$ wget -O - http://hudson-ci.org/debian/hudson-ci.org.key | sudo apt-key add -
$ echo "deb http://hudson-ci.org/debian binary/" >> /etc/apt/sources.list
$ apt-get update
$ aptitude install hudson
$ apt-get upgrade

That’s it! It’s already up and running on port 8080 using it’s own web server. Go ahead and pull it up in your browser.

As a test, let’s setup django-crm (a Caktus open-source community project) as our first Hudson job. Click “New Job”, type in a job name, click “Build a free-style software project”, and hit OK. django-crm contains a sample project that we’ll use to run the test suite. On the job configuration page, check Subversion in the Source Code Management section and type in the Repository URL:

Click Save, run the job by clicking “Build Now”, and check out the Console Output:

Started by user anonymous
Checking out a fresh workspace because /var/lib/hudson/jobs/django-crm/workspace/sample_project doesn't exist
Checking out http://django-crm.googlecode.com/svn/trunk/sample_project
A         manage.py
A         site_media
A         site_media/css
A         site_media/css/jquery.autocomplete.css
A         site_media/css/django-contactinfo.css
A         site_media/js
A         site_media/js/jquery-ui-1.7.2.custom.min.js
A         site_media/js/jquery-1.3.2.min.js
A         site_media/js/django-crm.js
A         site_media/js/jquery.autocomplete.min.js
...
Finished: SUCCESS

Cool, now let’s run some tests. Took keep things simple, let’s grab Django and a few dependencies using aptitude:

$ wget http://www.djangoproject.com/download/1.1.1/tarball/
$ tar xzvf Django-1.1.1.tar.gz
$ cd Django-1.1.1
$ sudo python setup.py install
$ aptitude install python-dev python-imaging python-setuptools python-pip

To run the tests, add an “Execute shell” build step in the Build section with this command:

#!/bin/bash -ex
cd sample_project
python manage.py test crm

Run the job again and look for the test results in the console output:

[workspace] $ /bin/sh -xe /tmp/hudson6670261053226891793.sh
+ cd sample_project
+ python manage.py test crm
...
Finished: SUCCESS

XML Test output

To integrate Hudson with the Django test suite, I used unittest-xml-reporting. Just “pip install unittest-xml-reporting” and add the following lines to your settings file:

TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.run_tests'
TEST_OUTPUT_VERBOSE = True
TEST_OUTPUT_DESCRIPTIONS = True
TEST_OUTPUT_DIR = 'xmlrunner'

Then check “Publish JUnit test result report” in the Post-build Actions section and add the path to the test XML output “sample_project/xmlrunner/*.xml”:

Run the job and you should see a new “Test Result” link in the navigation. Now you can view the test results right in your browser window.

Coverage

To add coverage reports, I used Ned Batchelder’s coverage.py (pip install coverage). Navigate to Hudson’s plugin manager (Hudson -> Manage Hudson -> Manage Plugins), install the Cobertura Plugin, and restart Hudson when prompted. Then modify your shell script like so:

#!/bin/bash -ex
cd sample_project
coverage run manage.py test crm
coverage xml --omit=/usr/

This will generate an XML coverage report in the working directory, so we just need to tell Hudson where to look for it. Check “Publish Cobertura Coverage Report” in the Post-build Actions section and enter the path to the report:

Run the build again and you should have access to a new “Coverage Report” link.

More to come…

This was just a simple example of getting Hudson setup with a Django project and I know a lot more can be done with Hudson (check out the large number of available plugins). The top items on my todo list are: see Hudson setup environments with virtualenv and pip, integrate more closely with the test suite (possibly using nose), check for PEP compliance, and setup build failure notifications. I hope to write more as I continue to setup our Hudson environment!

References

A few useful Hudson/Python/Django links I discovered while running through this setup:

• Django and Hudson and Bears, Oh My! (and twill, and figleaf)
• Setting up a python CI server with Hudson
• Django continuous integration with Hudson and Nose

Python and Django are tools we use on a daily basis to build fantastic web apps here at Caktus. I’m pleased to announce that Caktus is sending five developers–Colin, Alex, Mike, Mark, and myself–to PyCon 2010! PyCon is an annual gathering for users and developers of the open source Python programming language. This year the US conference is being held in Atlanta, GA. We’ll be driving down tomorrow (Thursday) from Chapel Hill, NC and staying for the conference weekend plus one day of the sprints.

Hope to see you there!

Django is a tool we use every day to build rock-solid web apps here at Caktus, and a development sprint is a concerted, focused period of time in which developers meet in the same space to get things done on a project.

We’re proud to annouce that Caktus is hosting a local Django development sprint in the Triangle (Raleigh, Durham, and Chapel Hill/Carrboro) area of North Carolina. The sprint will be held the weekend of December 12th and 13th in Carrboro Creative Coworking, and the purpose of this sprint will be to help finish features and push out bug fixes in preparation for the upcoming Django 1.2 release.

If you’re interested in attending, no previous experience contributing to Django is necessary and the sprint will be a great opportunity to start. Work on other open source Django-based projects is welcome too. For more information, check out the corresponding wiki page and don’t forget to register for the event.

We’ll be there to open the doors at 9am both days. Courtesy of our sponsors there will be free drinks, snacks, and lunch to go around. Hope to see you there!

Django’s ORM is great. It handles simple to fairly complex queries right out the box without having to write any SQL. If you need a complicated query, Django’s lets you use .extra(), and you can always fallback to raw SQL if need be, but then you lose the ORM’s bells and whistles. So it’s always nice to find solutions that allow you to tap into the ORM at different levels.

Recently, we were looking to perform a LEFT OUTER JOIN through a Many to Many relationship. For a lack of a better example, let’s use a Contact model (crm_contact), which has many Phones (crm_phones):

class Contact(models.Model):
    name = models.CharField(max_length=255)
    phones = models.ManyToManyField('Phone')
    addresses = models.ManyToManyField('Address')
 
class Phone(models.Model):
    number = models.CharField(max_length=16)

If we want to display each contact and corresponding phone numbers, looping through each contact in Contact.objects.all() and following the phones relationship will generate quite a few database queries (especially with a large contact table). select_related() doesn’t work in this scenario either, because it only supports Foreign Key relationships. We can use extra() to add a select parameter, but tables=['crm_phones'] will not generate a LEFT OUTER join type. We need to explicitly construct the JOIN.

DISCLAIMER: The following method does work, but should not be considered best practice. That is, there may be a better way to accomplish the same task (please comment if so!). But after sparse Google results for similar scenarios, I figured it’d at least be useful to post what we discovered.

After digging around in django.db.models.sql for a bit, we found BaseQuery.join in query.py. Among the possible arguments, the most important is connection, which is “a tuple (lhs, table, lhs_col, col) where ‘lhs’ is either an existing table alias or a table name. The join corresponds to the SQL equivalent of: lhs.lhs_col = table.col”. Further, the promote keyword argument will set the join type to be a LEFT OUTER JOIN.

Now we can explicitly setup the JOINs through crm_contact -> crm_contact_phones -> crm_phone:

contacts = Contact.objects.extra(
    select={'phone': 'crm_phone.number'}
).order_by('name')
 
# setup intial FROM clause
# OR contacts.query.get_initial_alias()
contacts.query.join((None, 'crm_contact', None, None))
 
# join to crm_contact_phones
connection = (
    'crm_contact',
    'crm_contact_phones',
    'id',
    'contact_id',
)
contacts.query.join(connection, promote=True)
 
# join to crm_phone
connection = (
    'crm_contact_phones',
    'crm_phone',
    'phone_id',
    'id',
)
contacts.query.join(connection, promote=True)

It’s a little verbose, but it accomplishes our goal. I used hardcoded table names/columns in the connection tuple to make it easier to follow, but we can also extract this information from the objects themselves:

contacts = Contact.objects.extra(
    select={'phone': 'crm_phone.number'}
).order_by('name')
 
# setup intial FROM clause
# OR contacts.query.get_initial_alias()
contacts.query.join((None, Contact._meta.db_table, None, None))
 
# join to crm_contact_phones
connection = (
    Contact._meta.db_table, # crm_contact
    Contact.phones.field.m2m_db_table(), # crm_contact_phones
    Contact._meta.pk.column, # etc...
    Contact.phones.field.m2m_column_name(),
)
contacts.query.join(connection, promote=True)
 
# join to crm_phone
connection = (
    Contact.phones.field.m2m_db_table(),
    Phone._meta.db_table,
    Contact.phones.field.m2m_reverse_name(),
    Phone._meta.pk.column,
)
contacts.query.join(connection, promote=True)

This results in a row for each phone number (Cartesian product), but we can print out each contact and corresponding phone numbers (with a single SQL statement) quickly in a template using {% ifchanged %}:

<h1>Contacts</h1>
 
{% for contact in contacts %}
    {% ifchanged contact.name %}
        <h2>{{ contact.name }}</h2>
    {% endifchanged %}
    <p>Phone: {{ contact.phone }}</p>
{% endfor %}

We’re pleased to announce that Caktus is looking for a developer to join our team on a contract basis!

What do we do? We build custom web applications for local and remote clients using a variety of open-source technologies. We are a small team founded in the Chapel Hill/Carrboro area (currently residing in Carrboro Creative Coworking) who believe in face-to-face contact and employ agile development techniques that emphasize teamwork and collaboration.

We’re looking for a strong software developer who enjoys working on a team and is excited to learn and experiment with new technologies. We do have a preference for local candidates, but will consider all submissions. Initial work will focus on maintaining small Django-powered websites. This will involve HTML/CSS (including converting Photoshop designs), Django Templates, and writing Unit Tests. Later work will involve creating and integrating Django apps into larger projects, deployment, and database work.

You will be working in Linux (Debian-flavor) production environments with Apache and WSGI. Python/Django experience is not required, but will be used on a daily basis. Relational database experience is a must. HTML/CSS and JavaScript experience are also a must, and jQuery is a plus.

If you’re interested in this position, please send us your resume, some example code, links to any open-source projects you’ve contributed to, and expected compensation. We’re excited to bring on a new team member!

At Caktus we’re big fans of reusing code. We leverage many open source projects–especially Django apps–to accomplish a variety of tasks. In addition, we’ve written quite a few pluggable apps over the paste two years that we reuse over and over again for different projects. As a way of giving back to the community, we’ve polished and released a portion of that code as open source ourselves. While some of the projects have been available on Google Code for awhile now, we just put together a consolidated list of open source Django projects on our web site to serve as a jumping off point for all the projects we like, we contributed to, and we created. Enjoy!

Django is a tool we use on a daily basis to build fantastic web apps here at Caktus, and DjangoCon is the annual conference for Django developers and other community members. We are proud to announce that Caktus Consulting Group, LLC is sponsoring DjangoCon 2009!

This year, the conference is being held the week of September 7th in the beautiful city of Portland, Oregon. Two Caktus partners, Colin and myself, will be attending. We hope to see you there!

In Django, a recursive many-to-many relationship is a ManyToManyField that points to the same model in which it’s defined (’self’). A symmetrical relationship is one in where, when a.contacts = [b], a is in b.contacts.

In changeset 8136, support for through models was added to the Django core. This allows you to create a many-to-many relationship that goes through a model of your choice:

class Contact(models.Model):
    contacts = models.ManyToManyField(
        'self',
        through='ContactRelationship',
        symmetrical=False,
    )
 
 
class ContactRelationship(models.Model):
    types = models.ManyToManyField(
        'RelationshipType',
        related_name='contact_relationships',
        blank=True,
    )
    from_contact = models.ForeignKey('Contact', related_name='from_contacts')
    to_contact = models.ForeignKey('Contact', related_name='to_contacts')
 
    class Meta:
        unique_together = ('from_contact', 'to_contact')

According to the Django Docs, you must set symmetrical=False for recursive many-to-many relationships. Sometimes–for a recent case in django-crm, for example–what you really want is a symmetrical, recursive many-to-many relationship.

The trick to getting this working is understanding what symmetrical=True actually does. From what we can tell after a brief look through the Django core, symmetrical=True is simply a utility that (a) creates a second, reverse relationship in the many-to-many table, and (b) hides the field in the related model (in this case the same model) from use by appending a ‘+’ to its name.

Since you normally have to create many-to-many relationships manually when a through model is specified, the solution is simply to leave symmetrical=False (otherwise it’ll raise an exception) and create the reverse relationship manually yourself via the through model:

crm.ContactRelationship.objects.create(
    from_contact=contact_a,
    to_contact=contact_b,
)
crm.ContactRelationship.objects.create(
    from_contact=contact_b,
    to_contact=contact_a,
)

Additionally, you’ll have to do a little cleanup to make sure both sides of the relationship are removed when one is removed, but otherwise this should achieve the same effect as setting symmetrical=True in other many-to-many relationships.

To hide the other side of the related manager, you can append a ‘+’ to the related_name, like so:

class Contact(models.Model):
    contacts = models.ManyToManyField(
        'self',
        through='ContactRelationship',
        symmetrical=False,
        related_name='related_contacts+',
    )

Good luck and feel free to comment with any questions!