There are some tools and apps which we use with almost all apps we write, and in particular which, we used for dinette. Here they are broken into useful during development, and (also) useful post development.

During Development

1. Ipython and ipdb
2. South
3. Django test utils
4. Django extensions
5. Django debug toolbar

Ipython and ipdb

Ipython is a enhanced shell for python. Ipdb similarly add extra capacity to the builtin pdb debugger. It is extremely convenient to drop into a ipython shell right where a complex piece of code is being hit.

from IPython.Shell import IPShellEmbed
ipython = IPShellEmbed()
ipython()

Of course this does not allow you to step through, so if you want to step through your code, you need to do

import ipdb
ipdb.set_trace()

Of course, you can just do pdb.set_trace(), but with tab completion and syntax highlighting, using ipdb is a no brainer.

South

South adds schema migration to Django. It has lot of advanced options, but just three command will get you along way.

Convert a normal app to use the south way.

./manage.py convert_to_south

Once you have made any changes to your apps model, write a new migration which can update the database to latest models.py

./manage.py startmigration appname nameofmigration --auto

Once you have written a migration, write a command to update the db.

./manage.py migrate appanme 

Django test utils

When you are running tests, a huge time sink is the time spent dropping and recreating tests. You might add just a single test, and the default Django testrunner will spend a huge time recreating the database. Enter manage.py quicktest which reuses databases between test runs. One of the side effects of this is that if you modify models.py between test runs you should manually recreate the database.

The tests (or whatever little of it exists) for Dinette were written using testutils’ testmaker.

Django extensions

Django extensions, (earlier Django command extensions) is a app which adds extra commands to manage.py. It has lot of goodies, but just ./manage.py shell_plus makes it worthwhile. (It imports all the models from all the apps automatically).

Get the full list at Github

Django debug toolbar

I do not personally use it but some people on our team swear by it.

After development

1. Django-pagination
2. sorl-thumbnails
3. django-compressor
4. Haystack

Django-pagination

I have decided on a standard way to build apps which need pagination. a. Build apps without pagination. b. The night before release spend an hour and add pagination to all pages. Really this is all you need

{% load pagination tags %}
{% autopaginate queryset %}
....
{% paginate %}

There is no change required to the views.

sorl-thumbnails

To thumbnail an image.

{% thumbnail image size %}

“Things should be as simple as possible, but not simpler.”

django-compressor

During development we work with multiple unminified js and css files. Django-compressor takes care of minifying and compressing it when we deploy. All we need to do is,

{% compress css %}
...
all css files
....
{% endcompress %}


{% compress js %}
...
all js files
....
{% endcompress %}

Haystack

Haystack is “modular search for Django”. It make writing search views as easy as writing the admin. You declaratively tell what fields you want to search on, and Haystack can take care of creating and updating the index, and providing a generic view to handle the search. Check a search view written using Haystack http://uswaretech.com/forum/search/?q=django

To everyone who contributed to these apps, thanks. Django development won’t be the same without these great tools. This post was inspired by Kevin Fricovsky’s post post, The apps that power Django-Mingus and is stolen about 50% from there. Thanks.

We build Amazing web apps, and we can build it for you. Contact us today.

Related posts:

1. Django for a Rails Developer
2. Doing things with Django models – aka – Django models tutorial
3. How to build a Facebook app in Django

What is bpython?

bpython is a fancy interface to the Python interpreter for Unix-like operating system.

says the bpython home page. It provides syntax highlighting, auto completion, auto-indentation and such stuff.

Unlike iPython, which implements then entire shell functions and emulates the standard python shell, and adds enhancements, bpython just adds features on top of the existing python shell functionality, using the curses module.

The “killer feature” of bpython is, as you can see from the image above, the IDE like prompting of the function parameters and the doc string of the function dynamically. I have always thought, what IntellijIDEA is to Java, IPython is to Python*. But bpython makes it more so, in terms of the code completion, which adds a lot of value to a ramping up developer.

The only Python IDE that provides Source Assistant and Go-To-Source functionality conveniently, is the commercial one, Wing IDE Professional. Even with that, since all the defined models are replaced (by using meta-classes) in the runtime, that source assistant is not perfect. Newer versions of Wing seems to claim to obtain data dynamically at the runtime, but its not always comfortable to write code, keeping the IDE in a breakpoint. But hey, bpython provides all these for free!

But I already use Ipython

You do? So do I. In fact, I am a Power Ipython user, I use all kinds of %magic functions: %whos, %logstart, %bookmark, %ed and sometimes use Ipython as an alternative even to bash: %cd, %ls, %ll. And even for doc testing: %doctest_mode, copy-pasting %hist -n or running some code module in the interpreter namespace: %run -i and searching code %psearch. You can also give any arbitrary bash(or your default shell) command by escaping it with !. Oh, ?, ?? and / are of course the starters.

In fact did you know, you could get to ipython shell within any arbitrary place within your django code? Just use the following:

from IPython.Shell import IPShellEmbed
ipython = IPShellEmbed()

and then call ipython() anywhere within your view, model, forms and you will be dropped to the shell. Its like ipdb.set_trace() only better (unless you want to step through, that is).

The Python readline bug

The Python version 2.6.4 (the version that is shipped with Ubuntu 9.10), introduced a readline bug that adds spaces after tab completion. This bug also affects the Ipython, as it uses the same readline module. If you spend a lot of time on the shell (if you use python, you must, right?), it is very annoying to backspace after each tab, all the time.

Although the bug got fixed pretty soon, it hasn’t yet made it to any release that ubuntu updates to. There are ways to workaround this problem, by fixing it at the python level and at the Ipython level, many of them are discussed on the corresponding Ipython bug

Using bpython with django

Now that you want to use bpython with django, either because you like the auto completion, or because you find the read line bug annoying (and don’t want/care-enough to patch your python locally), or you just want to try it, how to do it?

django manage.py shell unfortunately, drops only into the ipython shell (if you have ipython installed), and there is no straight forward way to get it to drop to bpython.

But there is still a way to use bpython with django. Just modify your ~/.bashrc to define a python startup environment variable

export PYTHONSTARTUP=~/.pythonrc

And within it, setup the django environment, that is, do it here manually the thing that python manage.py shell would do for you:

#.pythonrc
try:
    from django.core.management import setup_environ
    import settings
    setup_environ(settings)
    print 'imported django settings'
except:
    pass

This way, bpython or even just python on the command line, should import the django environment for you.

Importing all models automatically into the shell

But then, if you are possibly already used to python manage.py shell_plus (If you are not, you should be.) that is defined by django_command_extensions.

So while we are at setting up the django environment, lets just also import all the models into the shell namespace, so that it is convenient to work. Following is some ugly quick hack to get it done.

This can of course be improved upon. If you do it, just leave it in the comments! In fact it would be good if it also includes from startup.py import * in a try catch, so that, you can put some extra code into startup.py. Saving into startup.py from within a bpython shell is just a Ctrl+s anyway. That way each time you get to the shell, you can have the same expected environment; and it is quite easy to change that file.

*I know, I know, IPython doesn’t refactor code, nor build, nor does a million things that Intellij does, but Python is not Java and basically both intend to enhance developer productivity. And I spend a lot of time on the IPython shell and find it to be a tool just like Intellij is a tool, for Java.

Related posts:

1. Better Python package management using source and version control systems
2. Tools of Pro Django developer – aka What powers dinette and almost every app we write.

A quick django quiz. Answers available tomorrow. Get it as a text file (django-quiz) or on google docs or read below.

### Easy

1. You have a class defined as

    class Post(models.Model):
        name = models.CharField(max_length=100)
        is_active = models.BooleanField(default=False)

You create multiple objects of this type. If you do
Post.objects.get(is_active=False),

what exceptions is raised?

a. MultipleObjectsReturned
b. ManyObjectsReturned
c. SyntaxError
d. MultipleModelReturned
e. ManyModelReturned

2. Where is the function render_to_response defined?

a. django.views
b. django.shortcuts
c. django.templates
d. django.contrib.templates
e. django.contrib.shortcuts

3. What is the default name for the table created for model named Post
in application blog

a. post_blog
b. blog_post
c. postblog
d. blogpost
e. Postblog

4. How do you send a 302 redirect using django?

a. return HttpRedirectResponse()
b. return HttpResponseRedirect()
c. return HttpRedirectResponse(permanent=True)
d. return HttpResponseRedirect(permanent=True)
e. return HttpRedirectResponse

5. In django.contrib.auth, Users passwords are kept in what form?
a. Plain text
b. Hashed
c. Encrypted
d. Hashed then encrypted
3. Encrypted then hashed

6. Which of the following is correct way to find out if a request uses
HTTP POST method?

a. request.is_post() == True
b. request.METHOD == 'post'
c. request.METHOD == 'POST'
d. request.method == 'post'
e. request.method == 'POST'

7. Generic views have access to request context.

a. True
b. False
c. Default is True but can be set to False by GENERIC_REQUEST_CONTEXT in settings.
d. Default is False but can be set to True by GENERIC_REQUEST_CONTEXT in settings.
e. Can not be determined without extra information.

8. The current released version of Django is

a. 0.96
b. 1.0
c. 1.1
d. 1.2
e. Vyper logix 2.0

9. Which of these is a correct context_processor?

a.

def featured_posts(request):
    return Post.objects.filter(is_featured=True)

b.

def featured_posts(request):
    return {'featured_posts': Post.objects.filter(is_featured=True)}

c.

def featured_posts(request, response):
    return Post.objects.filter(is_featured=True)

d.

def featured_posts(request, response):
    return {'featured_posts': Post.objects.filter(is_featured=True)}

e.     
def featured_posts(request, response):
    response.write('featured_posts'= Post.objects.filter(is_featured=True))

10. Which of the following is a valid Middleware?

a.

class LatestPostMiddleware:
    def process_request(request):
        request.latest_post = Post.objects.latest()

b.

class LatestPostMiddleware(django.middlewares.BaseMiddleWare):
    def process_request(request):
        request.latest_post = Post.objects.latest()

c.

class LatestPostMiddleware(django.middlewares.BaseMiddleWare):
    def process_request(request):
        return {'latest_post': Post.objects.latest()}

d.

class LatestPostMiddleware():
    def process_request(request):
        return {'latest_post': Post.objects.latest()}

e.

class LatestPostMiddleware():
    def process_request(request):
        request.write('latest_post'= Post.objects.latest())


#Moderate

11. In the model below which line can be removed for the class to work

class Post(models.Model):
    name = models.CharField(max_length = 100)#Line 1
    desc = models.TextField()#Line 2
    post = models.ForeignKey(Blog)#Line 3
    blog = models.ForeignKey(Blog)#Line 4

a. Line 1
b. Line 2
c. Line 3
d. Line 4
e. Either of line 3 or 4

12. Who can access the admin site in Django?

a. user with is_super_user True
b. user with is_staff True
c. user with is_admin True
d. Either of a or b
e. Either of a, b, c

13. Which of the following code is closest to login_required decorator in Django?

a.

def login_required2(request):
    if request.user.is_authenticated():
        return True
    else:
        return False

b.

def login_required2(request, view_func):
    def check_login(request):
        if request.is_authenticated() and request.has_permissions(request.REQUIRED_PERMISSIONS):
            return view_function(request)
        else:
            return HttpResponseRedirect('/login/')
    return check_login(view_func)

c.

def login_required2(request, view_func):
    def check_login(request):
        if request.is_authenticated():
            return view_function(request)
        else:
            return HttpResponseRedirect('/login/')
    return check_login(view_func)

d.

def login_required2(view_func):
    def new_func(request):
        if request.user.is_authenticated():
            return view_func(request)
        else:
            return HttpResponseRedirect('/login/')
    return new_func


e.

def login_required2(view_func):
    def new_func(request):
        if request.user.is_authenticated() and request.has_permissions(request.REQUIRED_PERMISSIONS):
            return view_func(request)
        else:
            return HttpResponseRedirect('/login/')
    return new_func


14. Which of the following is a valid method to model Many to many relation ship in Django?

a.

class Foo(models.Model):
    bar = models.ManyToManyField(Bar)

class Bar(models.Model):
    foo = models.ManyToManyField(Foo)

b.

class Foo(models.Model):
    bar = models.ForeignKey(Bar)

class Bar(models.Model):
    foo = models.ForeignKey(Foo)

c.

class Foo(models.Model):
    bar = models.ManyToManyField(Bar)

class Bar(models.Model):
    pass

d. both B and C

e. All a, b, c

15. Which of the following is not included by default in TEMPLATE_CONTEXT_PROCESSORS

a. django.core.context_processors.auth
b. django.core.context_processors.debug
c. django.core.context_processors.i18n
d. django.core.context_processors.media
e. django.core.context_processors.request

16. Which of these is the currect way to validate uniqueness of a field named "slug" in a form
subclassing django.form.Forms.

a.

def clean(self):
    try:
        Post.objects.get(slug = self.cleaned_data['slug'])
        raise forms.Error(ValidationError, 'This slug already exists')
    except Post.DoesNotExist:
        return self.cleaned_data


b.

def clean(self):
    try:
        Post.objects.get(slug = self.cleaned_data['slug'])
        raise ValidationError('This slug already exists')
    except Post.DoesNotExist:
        return self.cleaned_data

c.

def clean_slug(self):
    try:
        Post.objects.get(slug = self.cleaned_data['slug'])
        raise forms.Error(ValidationError, 'This slug already exists')
    except Post.DoesNotExist:
        return self.cleaned_data


d.

def clean_slug(self):
    try:
        Post.objects.get(slug = self.cleaned_data['slug'])
        raise ValidationError('This slug already exists')
    except Post.DoesNotExist:
        return self.cleaned_data

e.

17. To add custom commands to your project setup, you need to,

1. Add management/command/commandname.py to your django app.
2. Add management/command/commandname.py to your django project.
3. Add command/commandname.py to your django app.
4. Add command/commandname.py to your django project.
5. Add management/command/manage.py to yout django project.

18. You want to enable caching for Django pages for AnonymousUser, which of these
is a valid way to do this.

a.


MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.CacheMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
)

b.

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.cache.CacheMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
)

c.

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.cache.CacheMiddleware',
)

d.

MIDDLEWARE_CLASSES = (
    'django.middleware.cache.CacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
)

e.

MIDDLEWARE_CLASSES = (
    'django.middleware.cache.CacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
)

19.

Consider the urlconf given below

    from django.conf.urls.defaults import *

    urlpatterns = patterns('news.views',
        (r'^articles/2003/$', 'year_archive_2003'),
        (r'^articles/(\d{4})/', 'year_archive'),
        (r'^articles/2003/(\d{2})/$', 'month_archive_2003'),
        (r'^articles/(\d{4})/(\d{2})/', 'month_archive'),
        (r'^articles/(\d{4})/10/', 'month_archive_october'),
    )
What view is called when accessing /articles/2003/10/

a. year_archive_2003
b. year_archive
c. month_archive_2003
d. month_archive
e. month_archive_october

20.

Consider the urlpatterns.

    urlpatterns = patterns('blogapp.views',
        (r'^list/(?P<year>\d{4})/(?P<month>[a-z]{3})/$','archive_month'),
    )

Which if the following is a valid view function which will be called on accesing
/list/2009/jun/

a. def archive_month(request):
        ...

b. def archive_month(request, *args):

c. def archive_month(request, **kwargs):

d. def  archive_month(request, year, month,):

e. Both c and d


# Hard

None yet

Follow us on twitter or subscribe to our blog

Related posts:

1. Doing things with Django forms
2. Understanding decorators
3. Using subdomains with Django

Django has three builtin template loaders which are used to get the templates for rendering.

TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.load_template_source',
    'django.template.loaders.app_directories.load_template_source',
#     'django.template.loaders.eggs.load_template_source',
)

Writing your template loader is a awfuly easy. It is a callable which

1. Returns a tuple of (openfile, filename) if it can find the template.
2. Raise TemplateDoesNotExist if the templates cannot be found.

The simplest template loader can be

from django.template import TemplateDoesNotExist
def load_template_source(template_name, template_dirs=None):
        try:
            return open(template_name).read(), template_name
        except IOError:
            raise TemplateDoesNotExist, template_name

if you put this to your template loaders directory,

TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.load_template_source',
    'django.template.loaders.app_directories.load_template_source',
    'dgrid.load_template_source'
#     'django.template.loaders.eggs.load_template_source',

You can do access a template using an absolute path.

In [6]: get_template('/home/shabda/abc.txt')
Out[6]: <django.template.Template object at 0x91c860c>

In [7]: temp = get_template('/home/shabda/abc.txt')

In [8]: temp
Out[8]: <django.template.Template object at 0x9358a0c>

In [9]: temp.render
Out[9]: <bound method Template.render of <django.template.Template object at 0x9358a0c>>

This is the first in the series of short and sweet Django posts we are going to do. You are interested, right. Do follow us on twitter or subscribe to our feed.

We build Amazing We Apps. Talk to us or email us at sales@uswaretech.com .

Related posts:

1. A response to Dropping Django

Brandon Bloom yesterday wrote an interesting post titled dropping Django. Despite a lot of hand waving(We needed a pragmatic template language to replace Django’s idealistic one.), it raises some valid questions, so here is a solution to some of them.

No support for hierarchical url creation.

The simplest representation of nested urls I can think of is a nested tuple. Lets represent, the urls for a simple app by,

>>> tree_urls = ('', 'list',
...     ('edit/', 'edit', ('auto/', 'edit_auto')),
...     ('^add/', 'add'),
...     ('delete/', 'delete', ('hard/', 'delete_hard'))
...     )

Guess what, urls.py is just a python module which excepts a variable names urlpatterns.

Which means it is very easy to write a function which converts this nested structure to flat, structure. Here is a quick attempt at that,

def merge(url):
    full_url=[]
    for i, el in enumerate(url):
        if i%2==0:    
            full_url.append(el)
    full_url = ''.join(full_url)
    return full_url

def combineflatten(seq):
    items= tuple(item for item in seq if not isinstance(item, tuple))
    yield items
    for item in seq:
            if isinstance(item, tuple):
                for yielded in combineflatten(item):
                    yield items+yielded

def generate_flat_urls(tree_urls):
    """
    >>> tree_urls = ('', 'list',
    ...     ('edit/', 'edit', ('auto/', 'edit_auto')),
    ...     ('^add/', 'add'),
    ...     ('delete/', 'delete', ('delete/', 'delete_hard'))
    ...     )

    >>> generate_flat_urls(tree_urls)
    [('^$', 'list'), ('^edit/$', 'edit'), ('^edit/auto/$', 'edit_auto'), ('^^add/$', 'add'), ('^delete/$', 'delete'), ('^delete/delete/$', 'delete_hard')]
    """
    return [('^%s$'%merge(el), el[-1]) for el in tuple(combineflatten(tree_urls))]

With this you can use hierarchical urls in urls.py as,

#All of urls.py
tree_urls = ('', 'list',
    ('edit/', 'edit', ('auto/', 'edit_auto')),
    ('^add/', 'add'),
    ('delete/', 'delete', ('delete/', 'delete_hard'))
    )

flat_urls = generate_flat_urls(tree_urls)

urlpatterns = patterns('app.views',
**flat_urls
)

No support for per user, per resource authorisation.

If you want to do this in a almost no-touching-the-existing-code way, replace all your render_to_response with,

def render_with_auth_check(request, payload, request_context, *args, **kwrags):
    for el in payload.itervalues():
        try:
            has_auth = el.check_auth(request.user)
            if not has_auth:
                raise AuthFailException
        except ValueError:
            pass #Not all objects have check_auth
    return render_to_response(request, payload, request_context, *args, **kwrags)

And enable this middleware,

class AuthFailHandlerMiddleware:
    def process_exception(self, request, exception):
        if type(exception) == AuthFailException:
            return render_to_response('accounts/login/', {}, RequestContext(request))

This assumes that all resources which are authorisation protected have a .check_auth, but I cant see any way round that in any other way as well.

A different, and more robust way would be to write custom managers for all resources, which need authorization.

class ResourceAuthManager(models.Manager):
    def get(self, user, *args, **kwargs):
        res = super(ResourceAuthManager, self).get(*args, **kwargs)
        try:
            has_auth = res.check_auth(request.user)
            if not has_auth:
                raise AuthFailException
        except ValueError:
            pass #Not all objects have check_auth

    ...

The Django templating language is too constrained.

Despite believing that Django templating language hits the sweet spot between, power and convenience to people who would be using it, I never understood this argument,

If you are using sqlalchemy with Django you lose the admin, what do you lose if you use Jinja?

In particular what do you lose by replacing render_to_response with this,

def render_jinja_to_response(template_name, payload):
    #This should probably go in settings.py
    from jinja2 import Environment
    env = Environment(loader=PackageLoader('app', 'templates'))

    template = env.get_template(template_name)
    response = template.render(**payload)
    return HttpResponse(response)

Extending authentication – In particular enable loggin in with email

Django authenticates against a wide range of backends, including Email, LDAP, Twitter and Facebook. While it is true that email backend doesn’t work with Admin login, Admin is meant for use by staff and superusers, so why cant we provide them usernames?

There are a few other questions raised, some of which I agree with(“Sadly, it[the admin app] struggles a little bit with nullable fields and is tricky to customize.”), and some which I dont, (“I will never write CSS by hand again.” – You shouldn’t be, someone else on your team should be doing that. ) But this line is interesting Personally, I've developed a moderate fear of the word "framework".

This is arguable, but the way all frameworks essentially help you is by providing Dependency Injection, (the Hollywood principle – Dont call me, I will call you). Django provides that in a lot of ways. (You write a middleware, and django calls it at appropriate places – Dont call me, I will call you), but still leaves something to be desired. My ideal framework would be one where models are POPO and views are functions which return strings . (Plain old python objects -like POJO). JPA does this correctly, but then JPA has the typing information, so maybe in python the only way to provide the required typing information is name=models.CharField(...). But the point being, we need better Dependency Inversion, not less of it. We have been dow that path earlier, and it is a much harder way to build complex systems.

Resources

1. This code on github
2. Discussion about converting the nested tuple to a flat one

My apologies if this post was aggressive, argumentative or disrespectful to the original author. We generally try to stay away from controversial posts on this blog. But I believe that this post raised some valid technical questions, and the intent of this post was to answer them. For Django to develop the best way, we need more of these kind of “What is wrong with Django” posts, not less, so my thanks to the original author.

Do you want to build Amazing Web Apps? We can help.

Related posts:

1. Django Request Response processing
2. Generating PDFs with Django
3. Django quiz

One of the new and most awaited features with Django 1.1 was aggregation. As usual, Django comes with a very comprehensive documentation for this. Here, I have tried to put this in how-to form.

Jump to howtos or Get source on Github.

Essentially, aggregations are nothing but a way to perform an operation on group of rows. In databases, they are represented by operators as sum, avg etc.

To do these operations Django added two new methods to querysets.

1. aggregate
2. annotate

When you are have a queryset you can do two operations on it,

1. Operate over the rowset to get a single value from it. (Such as sum of all salaries in the rowset)
2. Operate over the rowset to get a value for each row in the rowset via some related table.

The thing to notice is that option 1, will create one row from rowset, while option 2 will not change the number of rows in the rowset. If you are into analogies, you can think that option 1 is like a reduce and option 2 is like a map.

In sql terms, aggregate is a operation(SUM, AVG, MIN, MAX), without a group by, while annotate is a operation with a group by on rowset_table.id. (Unless explicitly overriden).

Ok enough talk, on to some actual work. Here is a fictional models.py representing a HRMS application. We will use this to see how to use aggreagtion to solve some common problems.

from django.db import models

class Department(models.Model):
    dept_name = models.CharField(max_length = 100)
    established_on = models.DateField()

    def __unicode__(self):
        return self.dept_name

class Level(models.Model):
    level_name = models.CharField(max_length = 100)
    pay_min = models.PositiveIntegerField()
    pay_max = models.PositiveIntegerField()

    def __unicode__(self):
        return self.level_name

class Employee(models.Model):
    emp_name = models.CharField(max_length = 100)
    department = models.ForeignKey(Department)
    level = models.ForeignKey(Level)
    reports_to = models.ForeignKey('self', null=True, blank=True)

    pay = models.PositiveIntegerField()
    joined_on = models.DateField()

class Leave(models.Model):
    employee = models.ForeignKey(Employee)
    leave_day = models.DateField()


"""
#Populate DB, so we can do some meaningful queries.
#Create Dept, Levels manually.
#Get the names file from http://dl.getdropbox.com/u/271935/djaggregations/names.pickle
#Or the whole sqlite database from http://dl.getdropbox.com/u/271935/djaggregations/bata.db
import random
from datetime import timedelta, date
import pickle
names = pickle.load(file('/home/shabda/names.pickle'))
for i in range(1000):
    emp = Employee()
    emp.name = random.choice(names)
    emp.department = random.choice(list(Department.objects.all()))
    emp.level = random.choice(Level.objects.all())
    try: emp.reports_to = random.choice(list(Employee.objects.filter(department=emp.department)))
    except:pass
    emp.pay = random.randint(emp.level.pay_min, emp.level.pay_max)
    emp.joined_on = emp.department.established_on + timedelta(days = random.randint(0, 200))
    emp.save()
"""

"""
employees = list(Employees.objects.all())
for i in range(100):
    employee = random.choice(employees)
    leave = Leave(employee = employee)
    leave.leave_day = date.today() - timedelta(days = random.randint(0, 365))
    leave.save()

"""

Find the total number of employees.

In sql you might want to do something like,

select count(id) from hrms_employee

Which becomes,

Employee.objects.all().aggregate(total=Count('id'))

If fact doing a connection.queries.pop() shows the exact query.

SELECT COUNT("hrms_employee"."id") AS "total" FROM "hrms_employee"

But wait, we have a builtin method already for that, Employee.objects.all().count(), so lets try something else.

Find the total pay of employees.

The CEO wants to find out what is the total salary expediture, this also converts the queryset to a single value, so we want to .aggregate here.

Employee.objects.all().aggregate(total_payment=Sum('pay'))

Gives you the total amount you are paying to your employees.

Find the total number of employees, per department.

Here we want a value per row in queryset, so we need to use aggregate here. Also, there would be one aggregated value per dpeartment, so we need to annotate Department queryset.

Department.objects.all().annotate(Count('employee'))

If you are only interested in name of department and employee count for it, you can do, Department.objects.values('dept_name').annotate(Count('employee'))

The sql is

SELECT "hrms_department"."dept_name", COUNT("hrms_employee"."id") AS "employee__count" FROM "hrms_department" LEFT OUTER JOIN "hrms_employee" ON ("hrms_department"."id" = "hrms_employee"."department_id") GROUP BY "hrms_department"."dept_name"

Find the total number of employees, for a specific department.

Here you could use either of .annotate or .aggregate,

Department.objects.filter(dept_name='Sales').values('dept_name').annotate(Count('employee'))
Department.objects.filter(dept_name='Sales').aggregate(Count('employee'))

If you see the SQLs, you will see that .annotate did a group by, while the .aggregate did not, but as there was only one row, group by had no effect.

Find the total number of employees, per department, per level

This time, we can not annotate either Department model, or the Level model, as we need to group by both department and level. So we will annotate on Employee

Employee.objects.values('department__dept_name', 'level__level_name').annotate(Count('id'))

This leads to the sql,

SELECT "hrms_department"."dept_name", "hrms_level"."level_name", COUNT("hrms_employee"."id") AS "id__count" FROM "hrms_employee" INNER JOIN "hrms_department" ON ("hrms_employee"."department_id" = "hrms_department"."id") INNER JOIN "hrms_level" ON ("hrms_employee"."level_id" = "hrms_level"."id") GROUP BY "hrms_department"."dept_name", "hrms_level"."level_name

Which combination of Employee and Deparments employes the most people

We can order on the annotated fields, so the last query is modified to,

Employee.objects.values('department__dept_name', 'level__level_name').annotate(employee_count = Count('id')).order_by('-employee_count')[:1]

Which employee name is the most common.

We can want to group by emp_name, so emp_name is added to values. After that we order on the annotated field and get the first element, to get the most common name.

Employee.objects.values('emp_name').annotate(name_count=Count('id')).order_by('-name_count')[:1]

This was a overview of how django annotations work. These remove a whole class of queries for which you had to use custom sql queries in the past.

Resources

1. Source on Github
2. sqlite file for this model to test
3. Aggregation on Django docs

Want to build a Django app? Talk to us

Related posts:

1. Doing things with Django models – aka – Django models tutorial
2. New tutorial – Building a search engine with Appengine and Yahoo
3. Dynamic forms with Django

When building public facing websites, spam is a real problem. Captcha has been teated as the first line of defence aginst this problem. If you must use captcha, here are some best practices working with them.

Can you do without one?

A lot of places captcha’s are put to filter spam in user generated comment. One of the largest sources of UGC is comments on wordpress blogs. They do not use a captcha, and instead pass all comments through Akismet to verify which comments are spam, and reject the spams. In many cases, such a system would work for you.

Can you use a simpler alternative?

Unless you run a site with millions of monthly users, the spammers are not going to write a bot specifically for youur site. So if you have a text question asking “what is 2 + 2″, the bots are not going to get past that, as the question is unique to your site. Of course, gmail/yahoo can not use this approach, as bots get written for them specifically, but for you site it might.

This proposed approach has been called SAPTCHA and anecdotal evidence suggests that it works very well in practice.

Prefer recaptcha

Recaptcha is an existing implementation of captcha, which allows you to plug in a captcha system, to your existing pages. Using this over home grown captcha’s offers advantages like

1. Its is battle tested, in being both secure against bots and having a level of usability.
2. It automatically provides and alternate means of validation. Which brings us to our third point.

Provide alternate means of validation to visually impaired people.

This might be in the form of an audio captcha, an option to email the administrator.

Do captcha validation only once

This seems so obvious, and yet many sites require you to complete captcha multiple times, in case of any error in the form. Once an user has completed captcha, the form must be displayed without captcha in case of errors.

If you use Django, these are already very easy to use with existing apps.

1. Django recaotcha
2. Django akismet
3. Python defensio

You should follow me on twitter here.

Related posts:

1. Parable of the Captcha – The futility of fighting automated spam with automated methods.

(If you are in a hurry, here is the fun part.

A few days ago, I was working with a nullable field, which wasn’t behaving as I expected. So I started a shell, and see how nulls compare.

In [1]: None

In [2]: None > 10

Out[2]: False

In [3]: None < 10

Out[3]: True

In [4]: None == None

Out[4]: True

So python is obviously broken, next steps, see the same thing in some other language. Lets try some other language, Javascript maybe?

//#Javascript
>>> null
null
>>> null == 10
false
>>> null > 10
false
>>> null < 10
true
>>> null == null
true

Ruby?

irb(main):001:0> nil
=> nil
irb(main):002:0> nil > 10
NoMethodError: undefined method >' for nil:NilClass
  from (irb):2
  from :0
irb(main):003:0> 10 > nil
ArgumentError: comparison of Fixnum with nil failed
  from (irb):3:in>'
  from (irb):3
  from :0
irb(main):004:0> 10 < nil
ArgumentError: comparison of Fixnum with nil failed
  from (irb):4:in ` 10 == nil
=> false

Comparision of how null behaves in different languages for

• null > 10
• null < 10
• null == 10
• null == null

All these are high level languages, and with exception of Java/Haskell/Sql are dynamically typed, so why is there this much difference in how None/null/nil as a type is handled?

As you can tell, I am very new to some of these languages, so if I have got your favorite language wrong. Please let me know in the comments and I will fix it.

Related posts:

1. The Rails and Django models layer Rosseta stone

This is announcement about our new work, Django design patterns, a ebook about, well, Django design patterns. (Well imagine that!). Here is the readme copied from there.

[Edit] Syntax highlighting and indentation preservation were totally brroken. Fixed now.

Django design patterns is a book about commonly occuring patterns in Django. Not patterns in Gof sense, but patterns in the sense, we work this way, and it works for us. For us it is a ~pattern of work~ sense.

At this point this is just a lot of brain dump from us.

The latest sources are always available from http://github.com/uswaretech/django-design-patterns/tree/master and latest html from http://djangodesignpatterns.uswaretech.net/

Please leave errata, feedback, critique as comments here.

This is still very much a work in progress, released in the spirit of release early, release often. Click here to get it, or fork it on Github

Related posts:

1. Django gotchas

Here is a quick tip. (Obvious if you work with Django for any length of time, but I have seen a few people who are not aware)

You can put debug trace import pdb; pdb.set_trace() in your code, and put it on the server. When you access this view from your local browser, the debug is still hit and you have a debug shell on your server where you can step through. (Obviously this works only if you ran the code via manage.py runserver. But manage.py runserver start the server to listen only on local connections. If you want to access remotely you need to run as,

python manage.py runserver 0.0.0.0:8000

Edit: As SmileyChris commented, a faster way is,

python manage.py runserver 0:8000

The 0.0.0.0 implies that remote connections are possible.

For me, this has been a lifesaver against those pesky bugs which show themselves only on the server, but not on the local machine.

Related posts:

1. Using Appengine with Django, why it is pretty much unusable