Porting a Django app to Jinja2 templates

Jinja2 has many nice features over native Django templates, one being a more strict sand-box security model and second template macros. Fortunately Django 1.8 introduced support for Jinja2 templates. In this post we will see how we ported our Django code base to Jinja2 templates.

DjaoDjin core product is an authentication and subscription hosted firewall. It integrates the Open Source projects djaodjin-signup, djaodjin-saas, djaodjin-pages and djaodjin-multitier. In all there were 80 template files for a total of 4.3K lines of HTML template code when we started the port. There was enough unknowns in moving to Jinja2 templates. We also wanted to move forward on other features in parallel while avoiding a merge nightmare. Since Django and Jinja2 have strong common roots, we settled on a a porting plan that involved re-writing the Django templates in a subset of the template language that was compatible with both Django and Jinja2. The idea was to do the code merge deployment in two steps

• 1. Deploy Django/Jinja2 compatible subset
• 2. Toggle Jinja2 template loaders

The easy bits

Undefined context variables

Context variables which are undefined in Django templates are silent. They are not in Jinja2. Undefined raise exceptions.

So far we had relied on the implicit rule "Missing from the context means False" in Django templates.

# Template code
{% if urls.profile %}
<a href="{{urls.profile}}">My Profile</a>
{% endif %}

To handle Jinja2 templates, we had to explicit set all variables used in the context.

{% csrf_token %} rewrite

The Django templates backend provides a convenient {% csrf_token %} context processor to insert a hidden field with the csrf_token in form elements. The Jinja2 templates backend always adds the csrf_token in the context by default.

-{% csrf_token %}
+<input type="hidden" name="csrfmiddlewaretoken" value="{{csrf_token}}">

{{ block.super }} vs. {{ super() }}

When you start to maintain a complex hierarchy of templates, super is really convenient.

$ cat base.html
{% block bodyscripts %}
<script type="text/javascript" charset="utf-8" src="/assets/base.js"></script>
{% endblock %}

$ cat index.html
{% extends "base.html" %}

{% block bodyscripts %}
{{ block.super }}
<script type="text/javascript" charset="utf-8" src="/assets/index.js"></script>
{% endblock %}

Unfortunately we haven't a way to get a compatible subset here so we have to rely on the old trick of defining sub-blocks that can be overridden deeper in the template hierarchy.

$ cat base.html
{% block bodyscripts %}
<script type="text/javascript" charset="utf-8" src="/assets/base.js"></script>
{% block base_bodyscripts %}
{% endblock %}
{% endblock %}

$ cat index.html
{% extends "base.html" %}

{% block base_bodyscripts %}
<script type="text/javascript" charset="utf-8" src="/assets/index.js"></script>
{% endblock %}

is_authenticated vs. is_authenticated()

request.user.is_authenticated is not defined as a @property so if we were to use it in a Jinja2 template, we need to add the extra parentheses, which would then render the template code incompatible with the Django backend. We resolve that by defining a context filter as such:

$ cat templatetags/authenticated_tags.py
...
@register.filter()
def is_authenticated(request):
    return hasattr(request, 'user') and request.user.is_authenticated()

We can then use the filter instead of calling the method directly.

{% if request.user|is_authenticated %}
...
{% endif %}

capfirst vs. capitalize

There are few filters that have different names in Django and Jinja2. The filter to capitalize a value is one of them. Following the previous pattern, we create a Django template filter with the Jinja2 name that calls the Django version.

$ cat templatetags/capitalize_tags.py
...
@register.filter()
def capitalize(text):
    return capfirst(text)

We also deal with get_*_display() methods using wrappers. In fact most functions and methods with zero or one parameters can be rewritten as filters to be compatible between Django and Jinja2 templates.

Moving Python code into Templates

What to do with crispy?

To render forms using a nice bootstrap layout, we used to use crispy_forms. There was going to be a significant effort to add features in crispy_forms to support both Django and Jinja2.

We were already feeling the constraints of crispy's philosophy of build template layout through Python code as we build the web product out by integrating multiple apps under a consistent UI theme.

Finally we got rid of the crispy_forms dependency and wrote forms as separate templates which are then included into the scaffolding.

$ cat index.html
<div>
{% include "_signup-form.html" %}
</div>

Front-end designers actually made a boost in productivity with this new implementation policy. They were also able to integrate AngularJS much better.

With good UX coverage, drift changes between the front-end templates and the backend code is caught early and easy to fix.

{% include with %} vs. {% macro %}

Macros are a very powerful construct in Jinja2 templates. We have been able to write macro logic that takes any Django form instance and generates an HTML layout entirely with macros in Jinja2 templates. It alleviated the use of crispy forms, as well as the murky .as_p, .as_ul Django defaults.

Unfortunately macros are not available in Django templates. It is though possible to simulate something similar with {% include with %} statements.

$ cat jinja2/_form.html
{% macro input_field(bound_field, hide_labels, extra_label_classes="", extra_control_classes="") -%}
...
{%- endmacro %}

{% for name, field in form.fields.items() %}
    {{ input_field(form[name], form.hide_labels) }}
{% endfor %}

$ cat django/_field.html
...
$ cat django/_form.html
{% for name, field in form.fields.items %}
    {% include "_field.html" with bound_field=form|get_bounded_field:name hide_labels=form.hide_labels %}
{% endfor %}

The form-handling templates is different for the Django and Jinja2 pipeline but for both, it is now entirely handled in template source files. With a few convention while passing the form instance into the context, every other templates can be written in a compatible subset.

$ cat index.html
...
{% include "_form.html" %}

Moving Template code into Python

{% if forloop.counter0|divisible_by:3 %} vs. {% if loop.index is divisibleby 3 %}

In the previous section, we converted layout code that was previously written in Python into template code compatible with Django and Jinja2. Here, to solve the compatibility issue, we did the opposite. We added attributes to elements of a collection passed into the template context.

$ cat views.py
...
    for index, plan in enumerate(context['plan_list']):
        if index % 3 == 0:
            setattr(plan, 'is_line_break', True)

$ cat pricing.html
    {% for plan in plan_list %}
        {% if plan.is_line_break %}
...
        {% endif %}
    {% endfor %}

URL patterns

One of the biggest Django/Jinja2 templates compatibility headaches deals with reversing url patterns and functions with more than one parameter (which can be dealt with filters).

# Django template code
{% url 'profile' 'bob' %}

# Jinja2 template code
{{ url('profile', 'bob') }}

We already extensively use Django Rest Framework serializers and location urls in serialized JSON objects that the AngularJS applications can use to retrieve further information. This to say front-end folks are already used to syntax like user.urls.profile.

We thus decided to move all URL reverse calls into Python code and pass them to templates through the context.

$ cat mixins.py
...
    context.update({
        'urls': {
            'organization': {
                'api_card': reverse('saas_api_card', args=(organization,)),
                'api_subscriptions': reverse(
                    'saas_api_subscription_list', args=(organization,)),
    }}})

$ cat checkout.html
...
{% block saas_bodyscripts %}
<script type="text/javascript">
    var reopenSaasApp = angular.module('saasApp');
    reopenSaasApp.constant('settings', {
        urls: {
            'organization': {
                'api_card': "{{urls.organization.api_card}}",
                'api_subscriptions': "{{urls.organization.api_subscriptions}}",
}}});
</script>

A side not here. The more we moved url reversal into the Python code, the more we hoisted it up into mixins. Since all DjaoDjin apps are meant to be used stand-alone or combined together, a good understanding of Python MRO becomes necessary to architecture Django apps as components. First time you get a TypeError: Error when calling the metaclass bases. Cannot create a consistent method resolution, you know you are pushing the limits.

Jinja2 rendering pipeline quirks

Django and Jinja2 filesystem loaders are different. You will deal with django.template.loaders.filesystem.Loader and jinja2.loaders.FileSystemLoader, which are totally unrelated. I would have expect the Jinja2 template pipeline to be a lot more integrated within Django's infrastructure. It is not (at this point).

To speed things up, Jinja2 uses a pre-compiled caching system for templates. The heart of it is in the Environment class:

$ cat jinja2/environment.py

    @internalcode
    def _load_template(self, name, globals):
...
        if self.cache is not None:
            template = self.cache.get(cache_key)
            if template is not None and (not self.auto_reload or
                                         template.is_up_to_date):
                return template

First, there is no regards to dimensioning caches. This was already an issue we bumped into when implementing multi-tier services in Django. It is best to make sure you size RAM on your server to handle storing all of the templates. The alternatives are to disable pre-compiled templates or restart your service regularly.