Three weeks ago we had a discussion about Ajax support in Django, which resulted in "Ajax and Django" wiki page. A short recap: it lists a vague goal, some general considerations, and possible strategies; it scratches the surface of existing implementations (mostly RoR), existing third-party toolkits (Mochikit/Dojo), and related RPC-style and REST-style services. No code was produced, no consensus was reached, but now it is a part of Django’s Version One roadmap.

Now three weeks later I assume that everybody had time to think about it. I want to move the whole discussion on to a more practical plane. Basically I want to propose an implementation. At least I’ll collect your thoughts, criticism, corrections, and, hopefully, blessings. Giving the speculative nature of this proposal, which is based solely on my experience, I decided against posting it in the wiki in its present form. The size of this proposal is not conductive to mail list posting. Hence it is here. Saddle up.

Rationale

I am a conservative pragmatic, who values the art of possible — don’t expect any outrages proposals. And like many programmers I am quite lazy — don’t expect glorious efforts required to implement my proposal. What I want from Django Ajax implementation is:

1. It should be realistically doable without major programming efforts.
2. It should be loosely coupled with Django.
3. Only Django-dependent parts should be implemented.
4. Simple things should be simple, complex things should be possible.

The reasoning behind my criteria and related conclusions are here:

• I don’t think we can afford spending months implementing Ajax library. While it is relatively easy to create, it is extremely difficult to debug under all possible client configurations. That’s why I have #1.
• Ajax library consists of two major parts: server-client interaction, and client-side programming. Django community is naturally positioned to push the former. That’s why I have #3. Additionally I don’t want to make critical changes to Django. It is as good as it gets even without Ajax.
• #4 is a sane rule of thumb. In practice it can mean that developers of some Django application may invest significant amount of time on client-side programming to implement their vision of user interface. Is Django community positioned to provide general JavaScript library? I don’t think so. Conclusion: the implementation should use some existing well supported Ajax library. We should augment it with Django-dependent parts and leave the rest to existing implementation.
• From #3 follows that all visual effects, DOM constructors, and other client-side helpers are out of scope of Django Ajax. They should be covered by Ajax library.
• #2 is common sense knowledge. Django and Ajax library should be loosely coupled. Such relationship allows both parts to evolve freely. We provide a bridge between them, nothing more. It satisfies #1 doing #3.
• The bridge should be high-level, rather than low-level. It ensures that we can formulate the big picture immediately without constructing intermediate layers while keeping the door open for other potential suitors (Ajax libraries). Let me remind that correct layering may depend on Ajax library as well.
• Saying that Django Ajax should be high-level subsystem, I don’t mean it should be a bloated monster with "well defined" interfaces described in brick-size book. I do mean we should concentrate on end user functionality instead of low-level technical details.

It leads to following results:

• I propose to use a widget paradigm: active client-side elements are self-contained widgets that interact with server side. While this interaction is widget-dependent, we should generalize it when possible.
• I propose to use Dojo. I hope that nobody surprised — I made my preference clear long time ago. Dojo’s model (general widget framework, AOP-inspired event system, I/O facilities, and so on) fits nicely in what we need. Dojo already has impressive collection of widgets, which can be adapted to use with Django.
• Choice of Dojo doesn’t prevent us from having a different implementation of Ajax component, but it will help us to formulate Ajax-related requirements, while producing practical results without major efforts.

Example

Let me give you a practical example. Coincidentally it is the poster child of Ajax-enabled Web 2.0 buzzword-infested applications: autosuggest/autocomplete box. Everybody uses it in some form and fashion: Google, Del.icio.us, Amazon, Flickr, and so on. RoR‘s demos feature it prominently. Let’s admit it: it can be a useful user interface feature. It allows narrowing down your choice from practically infinite (or just huge) pool of choices. It reduces GUI clutter significantly. End users love it. Let’s examine how Django can facilitate it.

Conceptually an autosuggest box accepts user’s keyboard input and presents valid choices in pop-up overlay, one of which can be immediately selected, or it can be refined further. Of course, in real life the widget waits for a short pause (usually ~0.3s) before submitting user’s input to server. Server interprets the input and returns a list of relevant objects, which is usually bounded. This upper bound prevents server overload, limits bandwidth, and ensures that only visible items will be shown.

How does Django fit in this picture? In most cases the server component (view function) will wrap existing basic lookup functions, namely get_list() and get_values(), with proper keyword parameters. Oversimplified code can look like that:

def autosuggest_view1(user_input):
    objects = friends.get_list(name__istartwith=user_input, limit=20)
    return render_to_response('autosuggest_template', {'objects': objects,})

Here is more flexible approach:

def autosuggest_view2(module, search_field, user_input,
                      search_method='istartswith', fields=None, limit=20):
    kwargs = {
        search_field + '__' + search_method: user_input,
        'fields': fields,
        'limit':  limit,
        }
    objects = module.get_values(**kwargs)
    return render_to_response('autosuggest_template', {'objects': objects,})

I hope you got the drift by now. Real implementation of autosuggest view will be a specialized generic view, which may take parameters directly from GET/POST applying them after some sanity check. If you want something custom, like merging objects from two different database tables, you can do it easily in your own code — there is nothing magical about autosuggest view.

Majority of data-driven widgets are like autosuggest widget: they work with a list of objects (1D data). The only difference is the criteria of selection: it can include additional restrictions, and ranges, limit/offset combinations. Well known examples are: all kind of lists (including hierarchical lists) and tables with predefined column structure. Admittedly they are the most popular widgets. It means that my autosuggest box example covers many possible scenarios and by implementing a compact set of Django views, we can cover almost all bases.

Now there is an interesting question: how to transmit data from client to server and back? I said above indirectly that client submits parameters using standard GET/POST mechanism. It is certainly the simplest way, which works in most cases. Complex cases will require JSON. We will decide what to use on per widget basis. Autosuggest widget will be happy with GET/POST.

I propose to use AHAH as a preferred form of server-client data transfer. AHAH is a fancy name for HTML fragments. In this case we can use existing Django template mechanism to present data in the most flexible way. For example, if in your autosuggest widget you want to show nicely formatted user’s information (picture, name, position, and department) instead of plain name — just do so using Django templates. If you want to output JSON or XML, you can do it with templates too — it is not the most efficient solution, but it is as flexible as possible. Of course we can provide better implementations for such cases.

How good is AHAH? Browsers support it natively. It is the fastest way to render HTML. It is the simplest way to render HTML. For example, RoR uses it for advanced autosuggest box demo. It sends information like that:

<ul class="contacts">
    <li class="contact">
        <div class="image"><img src="1.jpg"/></div>
        <div class="name">John Doe</div>
        <div class="email"><span class="informal">john@doe.com</span></div>
    </li>
    <li class="contact">
        <div class="image"><img src="2.jpg"/></div>
        <div class="name">Jane Doe</div>
        <div class="email"><span class="informal">jane@doe.com</span></div>
    </li>
</ul>

This information is formatted by CSS on client side. It has regular structure making it simple to do all kind of introspections from JavaScript.

Widgets

Now when you have an idea about proposed implementation of autosuggest widget, let’s talk about other widgets and what Django Ajax needs to do to support them. The list is based on my needs and backed by existing Dojo widgets. Obviously appearance and behavior of widgets should be customizable by developers. Feel free to add/remove widgets or argue merits. Widgets are listed without any particular order.

• Autosuggest widget. You should be sick of it by now. Dojo has it. We need to implement a custom data provider in JavaScript and tie it to Django Ajax views.
• Calendar/date picker. It is a companion widget to Django’s date-based generic views. User should be able to select year/month/day locally and to be transferred to appropriate view after selection. Django/developer should specify how to translate a date to URL. Additionally Django Ajax can supply a template tag to render widget’s representation in initial state. BTW, I wouldn’t mind to see "static" non-Ajax date picker as part of Django. Advanced implementation of date picker may be used to specify date intervals.
• Paginator. It is a companion widget to Django’s ObjectPaginator-based list/detail generic views. It can be used to update some portion of web page, without reloading the whole page. In a sense it is a variation of autosuggest widget.
• Scrollable tables. It is a fancy variation of Paginator widget. Dojo doesn’t have it at the moment but TurboAjax group is expected to donate it in some form (see it in action), Dojo’s biggest sponsor JotSpot is allocating resources to open source grid implementation (a-la Excel), and some volunteers porting OpenRico’s LiveGrid.
• Drag-and-drop widget, which allows transferring objects between several drop areas, or rearranging objects in a list. Basically it is similar to RoR’s implementation of Sortables, Draggables, and Droppables. For Django they look like variation of 1D data widget (yes, like autosuggest widget).
• Slideshow widget. It is yet another 1D data widget.
• Whole bunch of widgets, which have nothing to do with Django directly other than custom tags but a nice to have for useful GUI:

I think that’s enough for now. I would be happy, if we implement some reasonable subset first, for example, autosuggest, calendar/date picker, paginator, WYSIWYG editor, and drag-and-drop widgets. In any case let’s make Ajax version of Admin and let’s make it cool and easy to use.

I can try to adapt one Dojo widget at a time. Obviously it would be faster, if somebody can help me with widgets and/or server-side components. One thing I don’t want is to waste my time. So if somebody has better ideas or found flaws in my proposal, please voice it now in django-developers. If you support the proposal, please drop a line too.