Overview

The A-Team works on a variety of web applications, either as independent tools or as UIs to existing automation frameworks. Over time, we have experimented with a variety of web frameworks and systems. It now behooves us to take this experience, evaluate the frameworks and other building blocks, and converge on a set that will suffice for the majority of our new projects. Developers should be able to save time by using these blocks as starting points and thus concentrate on the "business logic" of their projects.

In short, the goal here is to exploit past experience to shorten development time of new web applications. However, we need to be flexible and thus shouldn't be enforcing strict policy, as there will be exceptions, and, since technology will continue to progress, we don't want to be locked into anything if at all possible.

Deadline/Deliverables

No deadline, but we would like a usable framework by the end of Q2. It will almost certainly be an on-going project after this, as we find other ways to improve our applications.

ATeam

• mcote

Dependencies

A new brasstacks, with a more recent OS, is strongly desirable, for the simple fact that nginx + WSGI (the most common Python-web server protocol) resists all attempts to be activated.

Major Tasks

• Examine the current web apps on brasstacks, backends and frontends (8-16h)
• Compare/contrast purpose of each tool with its backend and frontend and look for similarities--are some frameworks better/worse for particular types of applications? Create document with findings. (8h)
• Decide on a set of frameworks/modules/libraries for the most popular type of tool. Note that these do not have to work for *all* our future tools; rather, they should be sufficient for the majority. There will always be reasons to use other systems. Write this up with justifications. (8h)
• Document installation and usage of the UX system components, and templates/sample code where applicable (24-40h). The latter should include
  • Simple server code.
  • Examples of db connections from server
  • init.d script
  • nginx config

Notes

Server Architecture

For most apps, a growing consensus is that web.py provides most required functionality for relatively simple web tools. More complex applications might benefit from a heavier framework like Django, which will not be covered here (yet).

The standard described here will be client-side apps, that is, applications that have static HTML, CSS, and JS which use server-side REST APIs to update content. See http://metamarketsgroup.com/blog/node-js-and-the-javascript-age/ for a short summary.

The proposed layout is that the index file and other basic content resides at the root, with REST APIs being in a api/ directory. For example, the app 'toolfoo' might be found at http://server/toolfoo/ and its APIs at http://server/toolfoo/api/. Keeping a consistent approach will allow other applications to easily find and hook into our REST APIs should the need arise.

To keep scalability in mind, static content should be hosted by the main web server (nginx) and all dynamic calls passed through WSGI, FastCGI, or another gateway to the server-side app. Example nginx config, assuming static files are in the project's html/ directory and the application is being served on port 7890 (note that the order is important, to ensure the /api/ patch is matched first):

     location /toolfoo/api/ {
         fastcgi_read_timeout 300;
         fastcgi_param REQUEST_METHOD $request_method;
         fastcgi_param QUERY_STRING $query_string;
         fastcgi_param CONTENT_TYPE $content_type;
         fastcgi_param CONTENT_LENGTH $content_length;
         fastcgi_param GATEWAY_INTERFACE CGI/1.1;
         fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
         fastcgi_param REMOTE_ADDR $remote_addr;
         fastcgi_param REMOTE_PORT $remote_port;
         fastcgi_param SERVER_ADDR $server_addr;
         fastcgi_param SERVER_PORT $server_port;
         fastcgi_param SERVER_NAME $server_name;
         fastcgi_param SERVER_PROTOCOL $server_protocol;
         fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
         fastcgi_split_path_info ^(/toolfoo)(.*)$;
         fastcgi_param PATH_INFO $fastcgi_path_info;
         fastcgi_pass 127.0.0.1:7890;
     }

     location ~ ^/toolfoo/(.*)$ {
         alias /path/to/toolfoo/html/$1;
     }

For rapid development, we can use web.py itself with its built-in development web server to serve both the REST API and the static files. However, out of the box, web.py wants all static files in a static/ subdirectory, so we will need to hack around that. The advantage, however, of having the APIs be in a subdirectory off the application root means that everything is agnostic as to the absolute path on the server. All AJAX calls should be to the relative path "api/", so it doesn't matter if the full path is http://localhost:7890/ during development and http://server/toolfoo/ when deployed.

Databases

We have two recommended databases:

• MySQL: A Web standard and hosted locally.
• ElasticSearch: Very fast for searching, hosted by the metrics team--though we could talk about setting up our own local server, if needed.

Nonstandard dbs, like CouchDb, are not recommended unless there is a very good reason.

A random tip for database structures: store timestamps in UTC! It is much easier to deal with than most other time zones, particularly because there is no DST.

Client Architecture

Server-side templates versus AJAX-driven apps

This section is focussed on the client; however, the server is still involved as the source of the data and presentation layers.

There are two main methods of delivering functionality to the user in modern web applications:

1. server-side templates, which are filled in with the relevant data by the server before being served to the user
2. more-or-less static HTML, JS, and CSS files, which, via AJAX, fetch the data from the server after being served to the user

There can be some overlap, but generally web applications fall more into one category than the other.

There are several differentiating factors between these approaches.

1. Reliance on JavaScript. Applications using templates (method 1) do not need to rely on JavaScript. They may, and often do, use JavaScript to enhance the user experience, but since the data is present within the generated HTML, it is not strictly necessary. However in method 2, JavaScript is required, since the data must be fetched by the client browser through AJAX calls.
2. Dynamic updates. Method 2 can update the app with new data without requiring page refreshes, e.g. without user interaction. Apps strictly implementing method 1 cannot update their data without reloading.
3. Server load. Method 1 relies on the server generating the pages when they are loaded (depending on how caching is set up). Method 2 can better cache the static pages and transmit less data via raw formats like JSON.
4. Application richness. Method 2 avoids unnecessary page refreshes after form submission, to reload data, etc. Hence these apps can feel more like native apps. There is, however, always the danger of putting too much in the client and stressing the browser.
5. Reusable, bookmarkable, sensible URLs. It is much easier to create a good URL structure via method 1. It can be tricky to implement good URLs in method 2, however, since the state of the application can easily change without any page refreshes. HTML 5 has specified a way to fix this in the form of pushState() and related functions.

Thus, the major drawback of method 2 is the reliance on JavaScript. However most browsers, even on mobile devices, support JavaScript, so this is becoming less and less of a problem. Due to the separation between presentation (HTML/CSS) and data (REST APIs), JavaScript-less clients like the command-line utility curl can still fetch the data itself directly from the REST APIs. Furthermore, this approach enforces the creation of good APIs as the project is being designed and don't need to be tacked on later.

In conclusion, we should be using method 2 for most apps, unless there is a real need to support clients that do not have JavaScript capabilities. web.py can be used with either approach.

Client design

Nice separation of data and presentation can be ruined by a badly designed client. We should consider such concerns when designing the interaction between the HTML, CSS, and JavaScript.

In general, an "ad hoc" style should be avoided unless the project is extremely small. jQuery is very powerful but does not, on its own, enforce any organization and can easily lead to spaghetti code and a confusing mix of static and dynamically generated HTML. For projects of medium and high complexity, a more organized approach is desirable, as one would take with any software development. There are many, many libraries and frameworks available to do so, including

• MooTools
• backbone.js
• ...

[to be evaluated]

Regardless of which framework is chosen, as mentioned above a good URL structure should be implemented.

Application

The framework will provide an application, to create templates for new projects, and Python modules.

templeton init

This will create the basic structure for a new web app--server and html directories, basic server app and html template.

import templeton

Provides access to standard classes, e.g. JSON handler.