21
Oct
2008

Filed under: ajax, django, jquery

32 comments

AJAX upload progress bars with jQuery, Django and nginx

Update: the JavaScript originally used for the client side has been replaced with jquery-upload-progress, which works with Safari. I also took some time to clean up the Django upload handler and progress update view, so if you downloaded the demo app before December 4, 2008, you may want to get the latest version.

I've included upload progress bars on a couple of Django sites. Thanks to the new file handling in Django 1.0, a code snippet from djangosnippets.org, and jquery-upload-progress, it's really not difficult, but I haven't seen a complete example, so I put together a demo app showing how the pieces fit together. I'll also cover the case where you want a proxy to provide the upload progress tracking, without involving Django.

If you want to skip the lecture and download example code, go right ahead.

Background

The basic technique is simple: on the page containing your upload form is some JavaScript to generate a unique identifier for the upload request. When the form is submitted, that identifier is passed along with the rest of the data.

When the server starts receiving the POST request, it starts logging the bytes received for the identifier. While the file's being uploaded, the JavaScript on the upload page makes periodic requests asking for the upload progress, and updates a progress widget accordingly.

Client side

The client-side JavaScript is a straightforward application of jquery-upload-progress:

<script type="text/javascript" src="{{MEDIA_URL}}js/jquery.js"></script>
<script type="text/javascript" src="{{MEDIA_URL}}js/jquery.uploadProgress.js"></script>
<script type="text/javascript" charset="utf-8">
//<![CDATA[
$(document).ready(function() { 
    $(function() {
        $('#upload_form').uploadProgress({
            jqueryPath: "{{MEDIA_URL}}js/jquery.js",
            progressBar: '#progress_indicator',
            progressUrl: '{% url upload_progress %}',
            start: function() {
                $("#upload_form").hide();
                filename = $("#id_file").val().split(/[\/\\]/).pop();
                $("#progress_filename").html('Uploading ' + filename + "...");
                $("#progress_container").show();
            },
            uploadProgressPath: "{{MEDIA_URL}}js/jquery.uploadProgress.js",
            uploading: function(upload) {
                if (upload.percents == 100) {
                    window.clearTimeout(this.timer);
                    $("#progress_filename").html('Processing ' + filename + "...");
                } else {
                    $("#progress_filename").html('Uploading ' + filename + ': ' + upload.percents + '%');
                }
            },
            interval: 1000
        });
    });
});
//]]>
</script>

Add that to the bottom of your form page. For the actual progress bar, include this in your form page:

<div id="progress_container">
    <div id="progress_filename"></div>
    <div id="progress_bar">
        <div id="progress_indicator"></div>
    </div>
</div>
In your CSS, add something like this:
#progress_container {
    font-size: .9em;
    width: 100%;
    height: 1.25em;
    position: relative;
    margin: 3em 0;
    display: none;
}

#progress_filename {
    font-size: .9em;
    width: 100%;
}

#progress_bar {
    width: 100%;
    border: 1px solid #999;
}

#progress_indicator {
    background: #8a9;
    width: 0;
    height: 4px;
}

That should give you the progress bar. Now you just need progress reports....

Server side (Django)

On the server, you need two views: one to handle the upload form, and one to respond to progress requests. The first is completely standard; you don't have to do anything special for upload progress. Check the demo app, if you don't believe me. :^)

The magic happens in a special file upload handler, made possible by the new upload processing in Django 1.0. Again, the version here is derived from this snippet, with modifications to allow switching upload tracking back and forth between Django and nginx.

class UploadProgressCachedHandler(MemoryFileUploadHandler):
    """
    Tracks progress for file uploads.
    The http post request must contain a query parameter, 'X-Progress-ID',
    which should contain a unique string to identify the upload to be tracked.
    """

    def __init__(self, request=None):
        super(UploadProgressCachedHandler, self).__init__(request)
        self.progress_id = None
        self.cache_key = None

    def handle_raw_input(self, input_data, META, content_length, boundary, encoding=None):
        logger = logging.getLogger('uploaddemo.upload_handlers.UploadProgressCachedHandler.handle_raw_input')
        self.content_length = content_length
        if 'X-Progress-ID' in self.request.GET:
            self.progress_id = self.request.GET['X-Progress-ID']
        if self.progress_id:
            self.cache_key = "%s_%s" % (self.request.META['REMOTE_ADDR'], self.progress_id )
            cache.set(self.cache_key, {
                'state': 'uploading',
                'size': self.content_length,
                'received': 0
            })
            if settings.DEBUG:
                logger.debug('Initialized cache with %s' % cache.get(self.cache_key))
        else:
            logging.getLogger('UploadProgressCachedHandler').error("No progress ID.")

    def new_file(self, field_name, file_name, content_type, content_length, charset=None):
        pass

    def receive_data_chunk(self, raw_data, start):
        logger = logging.getLogger('uploaddemo.upload_handlers.UploadProgressCachedHandler.receive_data_chunk')
        if self.cache_key:
            data = cache.get(self.cache_key)
            if data:
                data['received'] += self.chunk_size
                cache.set(self.cache_key, data)
                if settings.DEBUG:
                    logger.debug('Updated cache with %s' % data)
        return raw_data

    def file_complete(self, file_size):
        pass

    def upload_complete(self):
        logger = logging.getLogger('uploaddemo.upload_handlers.UploadProgressCachedHandler.upload_complete')
        if settings.DEBUG:
            logger.debug('Upload complete for %s' % self.cache_key)
        if self.cache_key:
            cache.delete(self.cache_key)

What it does is take the generated tracking ID and as it's receiving the file, update the Django cache with the number of bytes received. The progress view just has to check the cache for that data:

def upload_progress(request):
    """
    Return JSON object with information about the progress of an upload.
    """
    progress_id = None
    if 'X-Progress-ID' in request.GET:
        progress_id = request.GET['X-Progress-ID']
    elif 'X-Progress-ID' in request.META:
        progress_id = request.META['X-Progress-ID']
    if progress_id:
        from django.utils import simplejson
        cache_key = "%s_%s" % (request.META['REMOTE_ADDR'], progress_id)
        data = cache.get(cache_key)
        json = simplejson.dumps(data)
        return HttpResponse(json)
    else:
        return HttpResponseBadRequest('Server Error: You must provide X-Progress-ID header or query param.')

The JSON returned is what the JavaScript uses to render the progress bar.

Server side (nginx)

While the Django upload handling got a lot better with the 1.0 release — the memory consumption is now controlled, and you have a lot more flexibility in storing the uploads — it may be better to let your web server shoulder the bulk of the work. That way it can deal with slow clients, and your hefty Django processes can be put to better use.

I use nginx and its mod_uploadprogress to do this. The technique is the same; the module tracks the upload progress and responds to AJAX requests for it. The difference is that your Django app doesn't get involved until the entire file is on the server, so the time it spends processing the upload is much less.

Here's the nginx config from the demo app in its entirety:

upload_progress uploaddemo 1m;

server {
    listen 80;
    server_name example.com;

    client_max_body_size 1000m;

    location ^~ /upload/progress {
        report_uploads uploaddemo;
    }

    location /media/ {
        alias /usr/local/django/test/uploaddemo/media/;
    }

    location / {
        fastcgi_pass                127.0.0.1:8080;
        fastcgi_pass_header         Authorization;          
        fastcgi_hide_header         X-Accel-Redirect;
        fastcgi_hide_header         X-Sendfile;
        fastcgi_intercept_errors    off;
        fastcgi_param               CONTENT_LENGTH          $content_length;
        fastcgi_param               CONTENT_TYPE            $content_type;
        fastcgi_param               PATH_INFO               $fastcgi_script_name;
        fastcgi_param               QUERY_STRING            $query_string;
        fastcgi_param               REMOTE_ADDR             $remote_addr;
        fastcgi_param               REQUEST_METHOD          $request_method;
        fastcgi_param               REQUEST_URI             $request_uri;
        fastcgi_param               SERVER_NAME             $server_name;
        fastcgi_param               SERVER_PORT             $server_port;
        fastcgi_param               SERVER_PROTOCOL         $server_protocol;
        track_uploads uploaddemo 30s;
    }
}

Last bit of blather

That gets you started with upload progress tracking. Of course, in a real application you're probably going to have more complex security concerns, and you'll probably flesh this out with better delivery of the uploaded content.

I handle all that with custom views that store uploads in a protected directory and delegate the actual file delivery to nginx via its X-Accel-Redirect feature; this could also be done with Apache via mod_xsendfile, or with lighttpd's X-Sendfile support.

Again, credit for the original Django upload progress handler goes to the snippet posted by ebartels on djangosnippets.org. All I've done here is put things together in the context of a complete app.

related files

Comments (32)

Liza Daly — 31 October 2008 9:00
Wow, this is awesome. I have exactly this stack running an app that accepts fairly large uploads. Thank you!
Jay States — 31 October 2008 11:26
Good look - I have been using nginx for about a month and find it nice. Now if nginx 0.7.x would work with mod_wsgi
john — 31 October 2008 13:49
@Liza: I'm happy to help such a cool site. Let me know if you have any problems getting it integrated.
john — 31 October 2008 13:57
@Jay: I haven't yet worked with 0.7, but there shouldn't be any difference proxying to Apache.

Or do you mean the nginx mod_wsgi? I have to confess I still don't see the point of that.
nick — 16 November 2008 8:37
If someone needs Safari/Opera working with this, checkout http://github.com/drogus/jquery-upload-progress/tree/master

For a quick check, you only need to change 2 lines in the js outcomment upload.state and make the uuid available to the update_progress url.
Works like a charm and thx for this blog post :)
Helped me alot !

Lakin Wecker — 23 November 2008 20:49
Hi,

I've been using a very similar method to yours, but I am using dojo instead of jQuery and I'm seeing the same behavior. Once the submit starts, Safari just doesn't submit any XHR requests to the server. :/

In your code above there is a small bug that you've probably never noticed due to safari not working, in the if statements that deal with the safari case, you refer to request.META in the first if statement when you probably meant to refer to request.GET:

elif 'X-Progress-Id' in request.GET:
# stupid Safari
progress_id = request.GET['X-Progress-Id']
elif 'X-Progress-Id' in request.META:
# stupid Safari
progress_id = request.META['X-Progress-Id']

john — 25 November 2008 9:49
@Lakin: Thanks. You're right, I'd given up on Safari. I really need to look into Nick's solution and get this updated with the result.

john — 4 December 2008 13:47
As noted at the top of the article, I've updated it and the demo app to use jquery-upload-progress. Safari now works, and the server side is a little cleaner. Thanks Nick, for letting me know about that.
Webagentur — 12 December 2008 12:16
Thank you ... this tutorial has me very helped.
alberto — 11 January 2009 22:31
This is awesome. I want to use this in a real application, but in the end of the article you said that there some security problems with this method. Which are that security problems?
john — 11 January 2009 23:19
Alberto: concerns, not problems. There's no security problem, as long as you intend for uploaded files to be public. The demo configuration presented here deposits uploads under MEDIA_ROOT.

If you don't want them shared with the world, you need to store them in a restricted location and control access to them in your Django views. For nginx, look into the "internal" directive:

http://wiki.codemongers.com/NginxHttpCoreModule#internal

With Apache and mod_xsendfile, you'd just make the upload directory "deny from all, allow from none".

Either way, the uploaded files will only be available if the Django response includes the appropriate header (X-Accel-Redirect or X-Sendfile).
sandeep — 4 March 2009 18:52
Hi, the link to uploaddemo.zip i dead. Any chance you can fix that? I'm having a difficult time getting all this to work properly. :(
Village Idiot — 5 March 2009 10:37
Will this work using the development server of Django? I think i read on one of those django snippet pages that this only works when hooked into Apache/Nginx. True? I've wired this up as the tutorial described but get no progress bar. Also second the previous commenters request for the uploaddemo.zip file. Please? Thanks!
john — 9 March 2009 6:39
The link to the ZIP file is fixed. Sorry about that.
john — 9 March 2009 6:51
@Village Idiot:

Django's development server isn't multithreaded, so it won't be able to process the progress update requests while it's receiving the upload.

I usually develop with something as close to production as I can get, while still allowing automatic reloading.

With Apache, set MaxRequestsPerChild to 1, or use mod_wsgi's reloading mechanisms (http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode).

Or try CherryPy (see http://lincolnloop.com/blog/2008/mar/25/serving-django-cherrypy/) or Spawning (http://www.eflorenzano.com/blog/post/spawning-django/), both of which I believe support automatic reloading and concurrent requests. You might find them a little easier to set up for development than Apache/mod_wsgi.
sandeep g. — 9 March 2009 12:11
finally got this all wired together. For others that might be having problems, a few things to check/fix.

I had to make a fix in jquery.uploadProgress.js:

upload.percents = Math.floor((upload.received / upload.total)*1000)/10;

i had to add "parseInt" around the strings.

upload.percents = Math.floor((parseInt(upload.received) / parseInt(upload.total))*1000)/10;

Also make sure that the parameter id string you are sending with the upload is the same one you are expecting to catch in the upload_progress view and javascript. The code examples bounce back and forth between "X-Progress-ID" and "HTTP_X_PROGRESS_ID"

When writing my file_handler, i had to add 'state': 'uploading' to the cache key since the uploadProgress.js seems to expect it.

And finally I was bitten by the lack of multi-thread in djangos dev server... so dont expect it to work until you try on a multi-threaded server setup!

thanks all for the help.
john — 9 March 2009 13:46
Sandeep, you should report the parseInt fix to the authors of jquery-upload-progress. I haven't seen that with IE, Firefox, or Safari on Windows or Mac.

The confusion between X-Progress-ID and HTTP_X_PROGRESS_ID is probably due to the fact that when you initiate the upload, you send X-Progress-ID as a parameter, but when jquery-upload-progress requests progress updates, it sends the ID in a header. That's why the upload progress view looks for request.META['HTTP_X_PROGRESS_ID'].

The handler in the post includes 'state': 'uploading' in the cached progress data. Or am I misunderstanding you here?
Antti Kaihola — 6 April 2009 7:32
john, looking at the current version of jquery.uploadProgress.js I don't see it sending the progress ID in a header but as a GET parameter:

[...]
jQuery.ajax({
type: "GET",
url: options.progressUrl + "?X-Progress-ID=" + options.uuid,
[...]
john — 6 April 2009 9:40
@akaihola: That explains the confusion; I was looking at my old version. The upload_progress view in the post has been changed to check both GET and META.

Thanks.
Skylar Saveland — 7 June 2009 22:51
Thanks, this is going to be great. I can put a bug to rest at http://mycogia.com/projects/project/improve_mycogia/tasks/?group_by=state
Viazanie diplomových prác — 14 June 2009 13:41
And what about Apache webserver?
john — 14 June 2009 18:36
What do you mean? You can either handle it in your Django view, or delegate it to Apache with mod_xsendfile, the same way you would with nginx or lighty.

I did have that working at one point, before switching over to nginx. This code has seen a few revisions since then, but it should work -- let me know if you try mod_xsendfile and it doesn't.
Damon Timm — 3 July 2009 9:55
Hey - thanks for the tutorial. I was playing with your demo site but am having trouble getting it to work on my shared host (site5).

I can't use memcache and django is being run by Apache through an fcgi handler.

Any thoughts on where I might turn to get it working ?

Thanks.
john — 4 July 2009 12:53
Damon, I haven't tried it through FastCGI, but you definitely have to have the cache. If you can't use memcache, you should be able to use one of the other backends.
Damon — 7 July 2009 13:26
Hi John - thanks for the reply. All right, I will keep expirimenting with one of the other backends. On your "test site" I did try to change it to "dummy:///" and "file:///" but neither made the progress bar move. I may have to try and expiriment with some others.

I wish something as simple as an upload progress bar was a little easier to implement!

Thanks again for your response.
Rafel — 5 August 2009 6:18
Hi, great tutorial! But it appears this error message:

File "/home/rafel/feina/Project3/project3/../project3/views.py", line 49, in <module>
class UploadProgressCachedHandler(MemoryFileUploadHandler):

NameError: name 'MemoryFileUploadHandler' is not defined

Do you have any idea of what it could be happening?

thanks in advanced!
john — 5 August 2009 10:32
Rafel, you're probably just missing an import; MemoryFileUploadHandler is in django.core.files.uploadhandler.
mattimck — 13 October 2009 9:22
So happy to find this tutorial, and i've been trying to use it along side django_imaging. I've almost had success, but once the upload progress bar appears, I start getting this javascript error every couple of seconds, until the file is uploaded, and it moves to the next page (i've replaced the site name with xxxxxxxx.com, obviously):

Error: upload is null
Source File: http://xxxxxxxx.com/site_media/media/imaging/jquery.uploadProgress.js
Line: 92

I'm using the "Server side (Django)" version, with Apache, mod_wsgi... and nginx serving up my media files.

I tried to use the nginx version of this, but couldn't figure out how to get mod_uploadprogress installed properly (I originally installed nginx with aptitude, but don't know how to add the uploadprogress stuff to that... then I tried uninstalling nginx and reinstalling using source, but had even more issues... so... i stuck with the Django only version in the end, since it got me seemingly close).

Am I doing something obviously wrong? Or am I completely misunderstanding the ability to use this without mod_uploadprogress and nginx?

Thanks! If anyone can help, I'd REALLY appreciate it!
nihongi — 1 November 2009 13:59
Hi mattimck,

I got the same error message, but finally resolved the problem and made the progress bar work.

Please make sure that your Python is work fine with memcached like this:
>>> import memcache
>>> mc = memcache.Client(['localhost:11211'])
>>> mc.flush_all()
>>> mc.set('key', 'value')
True
>>> mc.get('key')
'value'
john — 1 November 2009 14:55
I really need to find more time for this; I've been meaning to post a response to Matt's comment since we worked it out weeks ago.

He was using Django to handle upload progress, but still had nginx in front for handling static media. In that setup, nginx buffers the entire upload before passing it on to Apache/Django.

There's only a brief window where the upload progress might be stored in memcache and available to the JavaScript progress updater. So the upload works, but not the progress bar.

Matt's solution was to remove nginx from his setup. At this point, it's either that or use the nginx upload progress module. If nginx ever permits disabling buffering for uploads, this setup might work.


Michele — 1 December 2009 15:59
Hi John,

I have the same error as mattimck, yet I don't have nginx in front of apache. It's a simple apache + mod_wsgi seutp (debian lenny).
Upload works, but progress bar doesn't move : error in the firefox console is "upload is null".
Any ideas how I can proceed to debug this?

thanks,
Michele
nihongi — 3 December 2009 20:56
I made another simple sample app using this technique.

http://code.google.com/p/django-multi-file-uploader/

Please enjoy!

Comments have been turned off for this article, but you can always contact us about it.