July 18 2011 by Colin Copeland
We've been using RapidSMS, a Django-powered SMS framework, more and more frequently here at Caktus. It's evolved a lot over the past year-- from being reworked to feel more like a Django app, to merging the rapidsms-core-dev and rapidsms-contrib-apps-dev repositories into a single codebase (no more submodules!), to finally becoming installable via pypi. The "new core" is in a great state now and is much easier to work with. However, one particular aspect of RapidSMS, the route process, has always been complicated and confusing to deal with. Tobias began the conversation on this issue after returning from a 6-week long UNICEF project in Zambia. He summarized the route process like so:
- The route process as it currently stands is complicated; it includes a number of threads and the ways in which they interact is not always intuitive
- If the route process dies unexpectedly, all backends (and hence message processing) are brought offline
- Automated testing is difficult and inefficient, because the router (and all its threads) needs to be started/stopped for each test
The RapidSMS router is a globally instantiated object that routes incoming messages through each RapidSMS app and sends outgoing messages via installed backends. The run_router management command starts the router process and creates individual threads for each backend defined in the settings module. I'm not entirely certain as to why the route process was originally threaded, but I assume it was designed to more easily integrate blocking backends (like gsm) into RapidSMS. However, with the standardization of Kannel and SMS-based web services, like Twilio, both of which offload the low level communication work, I believe the threading aspect is now less important. So recently, in what started as a proof of concept, we began work on a decoupled router implementation called rapidsms-threadless-router. rapidsms-threadless-router provides a threadless_router app, which removes the threading functionality from the legacy Router class. Rather, all inbound requests are handled via the main HTTP thread. threadless_router attempts to:
- Make RapidSMS backends more Django-like. Use Django’s URL routing and views to handle inbound HTTP requests
- Remove clutter and complexity of route process and threaded backends
- Ease testing – no more threading or Queue modules slowing down tests
In comparison to the legacy route process, threadless_router handles all inbound and outbound backend communication from within the main HTTP thread. Each request creates a new router instance and no separate process or thread is created. This simplifies the Router class significantly. Additionally, threadless_router allows inbound messages to be easily passed off to an asynchronous task queue, such as Celery. Task queues allow message processing to be handled outside of the HTTP request/response cycle, which is perfect for SMS-based applications, as out of band responses are more than acceptable.
threadless_router is not, however, a drop-in replacement for the legacy router. Legacy backends will not work and as all routing is handled from within the HTTP thread, non-HTTP backends, such as pygsm, are not currently compatible with threadless_router. A simple wrapper around pygsm could be written to talk both to the modem and spin up a simple HTTP server to communicate with RapidSMS. This would decouple pygsm from RapidSMS and exist as it's own separate process. Integrating with supervisord would work great here too. Several contrib applications, such as httptester and scheduler, are also not compatible. We've bundled a new httptester as a replacement and celerybeat can be used to mimic the scheduler functionality. A full list of caveats can be found in the docs.
The full documentation for rapidsms-threadless-router, including installation instructions and examples, can be found on readthedocs.org. If you're already familiar with the internals of RapidSMS and would like to see examples of threadless backend implementations, I suggest reviewing the bundled http and httptester backends and our updated twilio backend.
I would like to mention that Nicolas Pottier and Eric Newcomer created rapidsms-httprouter, which also handles all messages within the main HTTP thread. The main difference between rapidsms-httprouter and rapidsms-threadless-router is that, while httprouter handles inbound messages in a Django view, it still starts up threads (for handling outgoing messages) like the current router (also from within a Django view). Make sure to check it out as well and let us know what you think!
December 29 2010 by Tobias McNulty
I recently returned from a 6 week trip in Malawi, where I was heavily involved in the implementation and deployment of Project Mwana, an Information and Communication Technology (ICT) project focused on Maternal and Newborn Child Health (MNCH). The project is currently running as a pilot in both Zambia and Malawi. This post is a fairly technical overview of what the project does and the way in which it was developed.
The project aims to facilitate several things, including (a) secure delivery of HIV (Dry Blood Spot, or DBS) test results from the lab to health clinics by SMS, which we’ve named “Results160″ (b) appointment reminders for newborn children, or “RemindMi” (Mi = mothers & infants), and (c) free-text “chat” for health clinic workers and Community Health Workers, to strengthen communication and patient tracing.
Source Code
The source code for the project, which is based on Django and RapidSMS, can be found on GitHub:
I updated the developer setup instructions fairly recently, so, if you’re developer interested in this project or line of work, you should be able to get a local copy up and running without too much trouble. If you try and do have any issues, please let me know!
Team Composition
In Zambia, we have 2 on-going local developers, 1 temporary lead developer, 1 on-going local project manager, and 1 on-going project mentor. The team is similar in Malawi, except we have 1 on-going local developer, 1 temporary lead developer (myself), 1 on-going local project manager, and 1 on-going project mentor.
Development Workflow
For this phase of the project we adopted “git flow” to help guide our development workflow, and I think it was a big success (more below). See the following links for more information:
Code Organization
apps/ - we tried to separate functionality into separate Django/RapidSMS apps as much as possible.backends/ - contains a RapidSMS backend for communicating with Kannel (an open source SMS gateway)locale/ - translation files for Bemba and Chichewarequirements/ - pip requirements files & corresponding tarballs (we found that checking in the tarballs was crucial for easily re-creating a development or production environment in low-bandwidth situations)malawi/ - Malawi-specific configuration files & codezambia/ - Zambia-specific configuration files & code
Settings Files
We took a strongly hierarchical approach to the settings files, to make it easy to share/override settings as necessary:
settings_project.py
\> malawi/settings_country.py
\-> malawi/settings_staging.py
\-> malawi/settings_production.py
\-> localsettings.py
\-> zambia/settings_country.py
\-> malawi/settings_staging.py
\-> malawi/settings_production.py
\-> localsettings.py
Each “sub” settings file simply imports from its “parent” at the top of the file, thereby allowing you to, for example, insert or append an app, add or remove a middleware, etc.
Apps
The project is divided up into the following main Django/RapidSMS apps: Results160 is implemented (mostly) in the “labresults” app, RemindMi in the “reminders” app, and the clinic chat in the “broadcast” app. The other apps support various pieces of functionality such as location management (”locations”), additions to the contact model (”contactsplus”), “stringcleaning,” and SMS printer integration (”tlcprinters”). I won’t pretend that they’re at all pluggable at the moment, but I’m sure that some of the more useful parts could be extracted & made independent, should the need arise.
Lessons learned
On the Malawi side, we learned a few things about building RapidSMS (or mobile health ICT projects in general) that I think are worth sharing:
(1) Regular meetings. Have quick meetings to review progress and plan next actions. Scrum type meetings really helped us review and quickly narrow down and fix things that we saw were not going too well before they became a problem.
(2) Pair Programming. Even though we were often pressed for time, pair programming proved very beneficial to the team and the project in general. Part of the mandate for the project is creating local capacity, and hands-on pair programming allowed for the newer or more junior local developers to both in gain a good overview of the codebase and learn from the more seasoned Python/Django developers on the team.
(3) Feature Branches. git-flow made them easy — which was key because we had two teams working simultaneously in two different countries (between which communication was difficult at times), and we wanted to share most of what we were doing while also shelter each other from potentially buggy commit of partially implemented features.
(4) Using a single code base for two deployments. It was difficult at times, when our two teams were changing different parts of the code and inadvertently breaking parts of it, but overall I think it was a big win & well worth the effort. We get the benefit of shared features & bug fixes, we don’t have to deal with maintaining separate forks, and it forces us to optimize our development workflow and make our code that much more configurable.
(5) Server environment. Get a public facing IP for your server and/or work location before beginning the implementation. We didn’t have one of these to start with in Zambia, while we did in Malawi. It makes life far, far easier for a number of reasons, including (a) it lets stakeholders get to the server (obviously), (b) it helps you be sure any connectivity issues are on the telco side, not yours, and (c) it lets you avoid corporate firewalls.
(6) SMS Gateway. If you have an SMS requirement, use Kannel. We started out with pygsm and our MultiTech modem would regularly get into a weird state where it was registered on the network & responded to AT commands, but no messages would come through. With other modems, the gsm backend didn’t delete messages from the SIM card, so it quickly filled up. After we switched to Kannel, we only had one case of downtime — and it was the RapidSMS route process, not Kannel, that was at fault. We also implemented the project on two different network carriers: Zain (now Airtel) via a GSM modem, and TNM (over the internet via SMPP). Kannel gave us a unified interface with which to interact with the two different backends. Kannel was also valuable as a “reference product” with which to test out the SMPP connection provided to us by TNM; as it turned out, when we couldn’t connect to it at first, the issue was on their end, not ours, and having a second opinion to back up our suspicions was key.
For more information, please see our page on Project Mwana or get in touch with a member of the Caktus team!
August 26 2010 by Tobias McNulty
DjangoCon 2010 is just around the corner, and I'm proud to announce that Caktus is sponsoring the conference again this year!
DjangoCon is the annual gathering of software developers who use the open source, Python-based Django web framework. We use the framework every day here at Caktus to create custom web applications and dynamic, content-rich web sites. Additionally, starting this year, we've put some of that knowledge to use extending and developing applications for the RapidSMS framework. For more information about why we use Django and think it's so great, check out our blog post titled Why Caktus Uses Django.
This year, the conference is being held again the week of September 6th in the beautiful city of Portland, Oregon. We've grown a little since last year at this time; it looks like 6 Caktus team members—Colin, Alex, Karen, Mark, Mike, and myself—will be attending the conference. We're positively thrilled to be going again this year and we hope to see you there!
June 07 2010 by Tobias McNulty
We're pleased to announce the release of the latest and greatest Caktus web presence yet. This edition features an enhanced services section and portfolio. Among other things, the new site demonstrates how our Django-based content management system can be used to connect related pages in customized, innovative ways.
In addition to Django web apps we've been building for years here at Caktus, we're now offering a related service in the mobile health development. Using RapidSMS, a communications framework built on Django, we build applications that add an SMS (text message) component to the standard Django stack. This allows non-standard users, such as clinic and community health workers, to interact with the system. Coupling high-tech and low-tech in this way lets us help remedy communication problems in places where Internet (and even power) are not widely available.
Check out the new site, and please let us know if you have any questions or comments!
