Cassandra Storage Cluster

One of the leading options for implementing the PiCL Storage Service is a Cassandra storage cluster. Cassandra provides a rich-enough data model and automatic cluster management, at the cost of eventual consistency and the vague fear that it will try to do something "clever" when you really don't want it to.

To get strong consistency back, we'd use a locking layer such as Zookeeper or memcached.

Basic principles:

• There is a single Cassandra storage node cluster fronted by the usual array of webhead machines.
• It uses a replication factor of 3, QUORUM reads and writes, and all notes live in a single datacenter.
• The webheads also have a shared ZooKeeper or memcached install, which they use to serialize operations on a per-user basis
• Cassandra is periodically snapshotted into S3 for extra durability.

From the POV of the webapp code, it's just talking to ZooKeeper and Cassandra Storage Node as abstract systems:

   +---------+          +-----------+
   | Web App |--------->| ZooKeeper |
   +---------+          +-----------+
           |
           |            +-----------+
           +----------->| Cassandra |
                        +-----------+

The fact that these are clustered, and membership may grow/shrink over time, should be transparent.

TODO: Try to use Route53 to provide consistent names for some of the nodes, to act as introducers even if the entire membership of the cluster has changed

The Cassandra cluster is periodically snapshotted into S3 for extra durability, using something like netflix's "priam" utility:

   https://github.com/Netflix/Priam

TODO We could also consider replicating just the Cassandra notes out to another DC for durability. This is expensive, but would allow us to more quickly stand up a replacement cluster if something awful happens in the one cluster's home datacenter:

   +-------------------------------+   +-----------------+
   |  US-East                      |   |  US-West        |
   |                               |   |                 |
   | +---------+    +-----------+  |   |                 |
   | | Web App |--->| ZooKeeper |  |   |  +-----------+  |
   | +---------+    +-----------+  |   |  | Cassandra |  |
   |        |                      |   |  +-----------+  |
   |        |                      |   |      ^          |
   |        |       +-----------+  |   |      |          |
   |        +------>| Cassandra |<-|---|------+          |
   |                +-----------+  |   |                 |
   |                               |   |                 |
   +-------------------------------+   +-----------------+