January 31, 2011

Django AJAX Tutorial Part 1 - adding email to newsletter

This is something I really wanted to write about since some time - using ajax with django. The cause is simple - internet is lacking good materials concerning this matter. And I remember having many problems with ajax in the beginning. That's why in the next few days I will show typical usage of ajax with django, like form validation, pagination and simple shoutbox.

Ok - the basics. First thing you need to understand - when talking about ajax here we mean a request which is sent from the browser by javascript, usually asynchronously ( meaning user can do other things while we're processing it). So to keep it simple - ajax is the 'request' we're using in our django views. And instead of reloading the whole page after user clicks some link or submits form we can first process it in the background and then refresh only the needed parts of the page. Sounds pretty facebookish doesn't it ? :) And there's no magic in it, really.

Let's start from the simplest possible example - we will allow user to add email to some newsletter. Here is tho code for model, url and form :

class NewsletterForm(forms.Form):
    email = forms.EmailField(label=_("E-mail address"))

class NewsletterEmails(models.Model):
    email = models.EmailField(_(u"E-mail address"),)

urlpatterns = patterns('',
    url(r'^newsletter_add/$', newsletter_add, name="newsletter_add"),

Nothing unexpected here I guess. Simple model, simple form with just one field and a link to our view (remember - even though we're not redirecting user to this link, we still need a way to call our view function processing the email).
Next comes the view function :

from django.http import HttpResponse
from django.template.loader import render_to_string
from django.utils import simplejson
from django.utils.functional import Promise
from django.utils.encoding import force_unicode
from models import NewsletterEmails
from forms import NewsletterForm

#here is some magic. We need to write custom JSON encoder, since sometimes
#serializing lazy strings can cause errors. 
#You need to write it only once and then can reuse in any app

class LazyEncoder(simplejson.JSONEncoder):
    """Encodes django's lazy i18n strings.
    def default(self, obj):
        if isinstance(obj, Promise):
            return force_unicode(obj)
        return obj

#here the view function starts
def newsletter_add(request):

    #as usual when processing forms we check if the method was POST/GET
    if request.method == "POST":

        #we get email address from request variables      
        adr = request.POST['email']
        e = None

        #we process address only if it's longer than 6 characters.
        #this assures us than we won't waste server time on
        #addresses like 'oeui' or 'u@u.u' 
        if len(adr) > 6:
            #create form instance and fill it with data. In this case it will only be the email address
            form = NewsletterForm(data=request.POST)
            #let django perform the validation
            if form.is_valid():

                #check if the email address is not already in our database
                    e = NewsletterEmails.objects.get(email = adr)
                    message = _(u"Email already added.")
                    type = "error"
                except NewsletterEmails.DoesNotExist:

                    if DoesNotExist exception was caught we add the address 'safe-mode' using try/catch
                        e = NewsletterEmails(email = adr)
                    except DoesNotExist:
                    message = _(u"Email added.")
                    type = "success"
                message = _(u"Bad address.")
                type = "error"                
            message = _(u"Too short address.")
            type = "error"

    #we don't need to do this but it's good practice to check the source of request. If it comes from ajax call we build the response.

    if request.is_ajax():
        #let built in simplejson package serialize our data. It means we transform everything to simple strings.
        result = simplejson.dumps({
            "message": message,
            "type": type,
        }, cls=LazyEncoder)

        #return the response
        return HttpResponse(result, mimetype='application/javascript')

Now the form. This is really basic so I won't say anything more about it apart of the fact that we explicitly set the action of our form to address of view processing addition of emails :

and finally the javascript part. I'm using jQuery javascript library here as well as jQuery's form plugin for processing forms. It simply adds method that takes all the values from form's fields and sends it to specified address (in this case it'll be "newsletter_add"). We could do this by ourselves, but that's not what this tutorial is about.

And basically that will be all for this part. As you can see the key to using AJAX with django is that our views must return serialized strings with JSON so that we can process them with javascript.

For the first time I have prepared a working example under this link :


as well as downloadable code :


If you have any questions feel free to ask in comments or on my twitter.

January 28, 2011

Synchronizing youtube user's videos filtered by tags

Today I will write about synchronizing your app with youtube and then filtering movies of specific user by tags. But let's get back to the beginning. I assume you already have django-syncr installed. I also assume that you're using video as attribute of some other object. Basic usage is pretty straightforward, so I'll proceed to some examples.

First the easiest one - we provide id of video as a string and as a result we receive Video instance :

from django-syncr.youtube import YoutubeSyncr

def sync_video_id(id):
    """ Synchronize single video by given YoutubeID    
    Takes youtube id as a string. Returns Video instance.   
    y = YoutubeSyncr()
        video = y.syncVideo(id)
        return video
        return video

That was easy. Let's try something more complicated then. Next there's a synchronization when provided url to video. Using python's built in urlparse module we divide url into components. Because my actual code needed to be compatible with version 2.4 of python, I'm using cgi module here (the same functions just stored in different place). Here's how this looks step by step. Everything is rather self-explanatory so I will not go through it line by line :

>>> url = 'http://www.youtube.com/watch?v=mPXxI1uyVEE&feature=rec-LGOUT-exp_fres
>>> url
>>> import urlparse
>>> url_data = urlparse.urlparse(url)
>>> url_data
ParseResult(scheme='http', netloc='www.youtube.com', path='/watch', params='', q
uery='v=mPXxI1uyVEE&feature=rec-LGOUT-exp_fresh+div-1r-3-HM', fragment='')
>>> query = urlparse.parse_qs(url_data[4])
>>> query
{'feature': ['rec-LGOUT-exp_fresh div-1r-3-HM'], 'v': ['mPXxI1uyVEE']}
>>> id = query["v"][0]
>>> id

and that's how we will do this in django :

from django-syncr.youtube import YoutubeSyncr

def sync_video_url(url):
    """ synchronize single video by it's url    
    Takes video url as a string, returns Video object.

    import urlparse
    url_data = urlparse.urlparse(url)
        query = cgi.parse_qs(url_data[4])
        query = urlparse.parse_qs(url_data[4])
    id = query["v"][0]
    y = YoutubeSyncr()
        return video = y.syncVideo(id)
        return none

When you're finished warm up It's time for more complex functionality. Now we will sync videos of specific users meeting specified tags requirements. First the core function that will do the work here. Because I tried to make it reusable django's ContentType module is used. :

from django.contrib.contenttypes import generic
from django.contrib.contenttypes.models import ContentType
from django-syncr.youtube import YoutubeSyncr

def sync_video_user(parent,model_name,user,tags):
    """ Sync all videos of user matching given tags pattern .
    Takes parent object, object to sync videos (as a string), string with username and string with tags.
    Returns list of synchronized videos. Takes care of already existing videos as well as prevents duplication of slugs.
    from django.template.defaultfilters import slugify
    #first we need to get the actual model for object holding video.
    model_class = ContentType.objects.get(model=model_name).model_class()
    #we create a queryset for searching for duplicates of video we're currently worknig on
    queryset = model_class._default_manager.all()
    #parse string of tags to get proper validating url string
    fmt_tags = parse_tags(tags)

    #request for youtube feed
    feed = get_youtube_feed_url(user, fmt_tags)
    #search for videos parsing returned feed
    sync = search_youtube(feed)

    #create a temporary instance (notice not using save() anywhere here) to get object methods available as well as queryset
    instance = model_class()

    for vid in sync:
        #if our queryset does not already contain object with this video id
        if not queryset.filter(**{"yt_video_id": vid.video_id}):

            #check if slug is free
            free = try_slug(instance, slugify(vid.title))
            if not free:
                #if not use function which we had already discussed some time ago - unqiue_slugify
                free = unique_slugify(instance, slugify(vid.title))
                free = slugify(vid.title)

            #create new object
            new = model_class(parent=parent,
                               name = vid.title,
                               publication_date=vid.published, active=True,

    return 1

Now that we have the main part of the code, we can look on the smaller pieces :

def parse_tags(tags):
    Parse received string of tags to tags list. Then append them to yt search API address.
    Return path to search api.    

    tags_list = tags.split(',')
    tags_list = [tag.strip() for tag in tags_list]
    tags_list = [re.sub(r"\s+", "+", tag) for tag in tags_list
    parse_string = '&category=%s&v=2' % '%2C+'.join(tag.strip() for tag in tags_list)

    return parse_string

Not much magic going on here. We split string of tags and use simple regex to get single tags from it (tags are either separated by commas or more than two whitespaces. Multi-words tags allowed). We then put all tags to query url in the proper form.

Next function adds our tags to the rest of the query string containing youtube username :

def get_youtube_feed_url(user, url):
    """ Add username to youtube feed address.
    feed = '/feeds/api/videos?author=%s&alt=rss' % user
    feed += url
    return feed

Finally when we have the whole query url we can perform the search using youtube api :

def search_youtube(path):
    """ Search videos on youtube
    If path is given, connects to youtube feed returns any videos matching query.
    All videos are then synched.
    if path:
        import feedparser
        import urlparse
        YTSearchFeed = feedparser.parse("http://gdata.youtube.com" + path)
        videos = []
        for yt in YTSearchFeed.entries:

            #the only new part is here. Because returned feed is a really complex json file, we search for 'link' keys that store url to our desired videos       
            url_data = urlparse.urlparse(yt['link'])
                query = cgi.parse_qs(url_data[4])
                query = urlparse.parse_qs(url_data[4])
            id = query["v"][0]
        synched = []
        for video in videos:
                sync = sync_video_id(video)
        return synched
        return []

And basically we're finished.

January 9, 2011

Using ProxyPass with django project

Once again I need to postpone my post about django-syncr and youtube since I came across interesting problem recently. Here's the deal : user has his django app running on some server under some link. Application itself is visible under different link. So apache conf got 'ProxyPass' and 'ProxyPassReverse' added. All seemed to work fine apart of links created with '{% url %}' templatetag (and from what I was able to test also get_absolute_urls could cause problems depending on the method used to create them). Url tag was adding original django application name to the link. Because I'm no good in setting apache I've came up with different solution. After skimming through django/template/defaulttags and django/core/urlesolvers I've realized that rewriting both 'reverse' function and 'url' tag would be tedious and probably unnecessary. Instead I've just added this 'try' block that replaces first unwanted occurence of our app name (which is defined in settings under PROJECT_NAME variable) with empty sign to 'url' templatetag and saved it as custom tag named 'urlc'. Not sure if overriding default tags works, gotta check it some day.

                if self.asvar is None:
                    raise e
        #code starts here
            url = url.replace(settings.PROJECT_NAME+'/', '', 1)
        #code ends here    

        if self.asvar:
            context[self.asvar] = url
            return ''

Not sure if it works for OP, but worked at my test config without bigger problems.

January 8, 2011

Provide unique slugs for model class

Sorry my dear readers for this long break between posts, but I have had lots of work lately finishing previous jobs. Recently I've shown you how to synchronize your app with Twitter and also parse list of tweets changing hash tags to proper links. Today I wanted to show you how to synchronize application with youtube but for this purpose I need a way to provide unique slugs for models. So here's the whole code which I'll comment below :

import re

from django.template.defaultfilters import slugify

def try_slug(instance, slug):
    """Check if the slug is free for corresponding model instance
    queryset = instance.__class__._default_manager.all()
    if not queryset.filter(**{"slug": slug}):
        return True
        return False

def unique_slugify(instance, value, slug_field_name='slug', queryset=None, slug_separator='-'):
    """Create unique slug
    Creates unique slug across model class.
    slug_field = instance._meta.get_field(slug_field_name)
    slug_len = slug_field.max_length

    slug = slugify(value)
    slug = slug[:slug_len]
    slug = _slug_strip(slug, slug_separator)
    original_slug = slug
    if queryset is None:
        queryset = instance.__class__._default_manager.all()
    if instance.pk:
        queryset = queryset.exclude(pk=instance.pk)
    next = 2
    while not slug or queryset.filter(**{slug_field_name: slug}):
        slug = original_slug
        end = '%s%s' % (slug_separator, next)
        if slug_len and len(slug) + len(end) > slug_len:
            slug = slug[:slug_len-len(end)]
            slug = _slug_strip(slug, slug_separator)
        slug = '%s%s' % (slug, end)
        next += 1
    return slug

def _slug_strip(value, separator='-'):
    Cleans up a slug by removing slug separator characters that occur at the
    beginning or end of a slug.
    separator = separator or ''
    if separator == '-' or not separator:
        re_sep = '-'
        re_sep = '(?:-|%s)' % re.escape(separator)

    if separator != re_sep:
        value = re.sub('%s+' % re_sep, separator, value)

    if separator:
        if separator != '-':
            re_sep = re.escape(separator)
        value = re.sub(r'^%s+|%s+$' % (re_sep, re_sep), '', value)
    return value

So now some explanations to the code. First function gives us possibility to check if slug is free for specific model class. It takes object instance and slug and returns boolean value. Inside the code we take default 'all' manager of the model and filter queryset with slug.
Second function is the key to solving our problem. The unique_slugify function takes shown parameters. At first we check the max length of the slug field, so that we won't exceed its value. Then we slugify our slug with built in django function, cut it if needed and strip of sepparators different than '-' or 'separator' parameter received by function. Next step is to get the queryset on which we will be working, excluding our model instance. Finally the fun begins. In our while loop condition we check existence of newly created slug or result of filtering our queryset. If both are none we take the separator and add next integer to it.
If the length of concatenated slug and identifier is too long, we cut it off. Finally we're creating slug that will be used in while loop condition filter.
Last function is stripping our slug by separators appearing at the beginning and end of slug. Because it uses many regexes I will leave in your hands work of decyphering it :)

And now for some short sample usage :

free = try_slug(self.instance, self.cleaned_data['slug'])
if not free:
    self.cleaned_data['slug'] = unique_slugify(self.instance, self.cleaned_data['slug'])