Raindrop/Milestone Planning

Current Milestone

See all Milestones

Arbutus

Hosted Goal

Our Q2 goal for Raindrop is having a hosted Raindrop that is in restricted Alpha. The goal of this hosted version is to get feedback on the design. It will be a very experimental alpha, users should expect volatile behavior.

On the path to get to a restricted alpha, we need to bootstrap to a dogfoodable, hosted version so we can start to analyze cost and get the UI polished as we use it daily for our own needs.

To help contain the cost, refine the mission for the Raindrop product:

Inflow's target experience helps users process new conversations and people. To that end, to contain costs and help performance, we only will do last 2 weeks worth of messages.

Later we could store more messages, but that would increase the cost for the user. More study will be done after we have more experience with the 2 week limit.

Immediate primary goals: reduce the big risks of hosting Raindrop:

• Platform Costs (performance/hosting costs)
• User Experience

How to get there for both items?

• Bootstrap hosted version for dogfooding by team (See Sequoia)
• Set performance goals
• Measure current version
• Build in feedback loop
• Use measured data to do changes

Performance Goals

Platform

• Load initial UI under 2 seconds
• Each API call under 2 seconds
• Expand group widget, 1 second
• View conversations for a group (like mailing lists) 2 seconds
• Time to process a message after fetching from message provider?
• Come up with a cost per user per month target. May need real data first.

User Experience

• Sense of relief?
• Be able to use on daily basis?
• Ease of processing new mail?

Measure Current Version

Platform

• Measure cost of running Jean's account Gozer
• Disable/enable some features/extensions to see the effect on cost Mark to expand?

User Experience

• List out the broken pieces, things that must be working to get outside people to try limited experience test. Bryan and Andy

Feedback Loop

Platform

• As we do changes, how do we measure increase in performance/affect on per user cost? Mark/Gozer
  • Why not build a simple high level usage plan (initial load, click n messages, delete something, etc) and translate that to a list of URLs that would need to be fetched. From that, building a simple, automated measurment of how much time it takes to complete that could for a nice baseline number to follow around.
  • Also, it might be time to start thinking about Continuous Integration so we can track changes back to code changes.

User Experience

• Identify up to 10 people for initial feedback test?
• How to to set the tone for the feedback?
• How to collect the feedback?
• How to feed new "tests" to the initial users?

Changes Based on Measurements

This section is likely to be filled in later after we have more data, maybe as part of another milestone. Some initial thoughts from Bryan (but need real data to confirm effectiveness):

• From Bryan: put summary docs in a different DB, and use custom views on it, to replace the API layer. Benefits, easier to play with for developer writing front end extensions, may help performance?
• Reduce the flexibility of the backend schemas.

User reqs

• Only one gmail account and one twitter account allowed to be registered.
• Firefox or Webkit desktop browser only.
• English only.

Explicit non-goals

• Contact/relationship design will likely not be included. We probably need feedback/metrics from initial alpha before doing this.
• No tagging.
• No full text search. Push off to google if possible.
• If we have time, do reply/compose. If it has bugs or is not designed well then that is a problem. It will likely not be included. When the work is done, this is the order of importance:
  • Twitter stuff first.
  • Email, plain text
  • Email fancy style.
• No mobile, strictly desktop browser.
• Developer tools will not be improved, the goal is for design feedback. But we should invite developers so they can try it and get interested in the code.

High level pieces

• Minimal Web Site for landing page. Andy
  • Point users to the blog for news.
  • It will not take requests for invites.
  • When landed on with the right URL code a "Create Account" button appears
    • Create Account button opens a popup window for logging in
• See Weave project for possible templates on privacy/terms of use etc... Rafael?

• Invite system: Only do it by hand, we choose the people manually. Need to work with Gozer on how to tie invite links to people's accounts. Gozer?

• Invite link goes to the Account page. Restrict account setup to the invite email name.
  • Invite Email text Andy/Bryan

• Create Account page. Needs the following: Bryan/Andy
  • Explanation
    • Not for sensitive/confidential materials. We will do our best but this is an Alpha. Link to tech on a wiki page.
    • What Raindrop can do (mostly read-only for now)
    • What is required of you
      • Participate, File Issues, Look for problems in layout and rendering
        • You can file new ideas but we're looking for what isn't working, not what isn't there
    • Last N days of email only (explain that in explanation).
  • Accounts
    • Enter Account here and here. You may delete them later
  • Only Gmail, twitter one account each.
  • "We'll email you when its ready" - no waiting UI is needed.
  • ( Start ) buttons starts the run-raindrop
    • Check your email in a couple minutes when we are ready
  • Popup window closes when ( Close ) button is clicked
  • At this point the account page only allows the user to delete their account
    • There is no logout or change of email addresses

• Send email when the run-raindrop is done! Mark?
  • When run-raindrop runs, at the end of each run, check for a rd.last_sync schema.
  • If that schema does not exist, write it with the current time stamp and send email using the gmail credentials for the user. So the user ends up getting an email from themself.
  • If that schema already exists, update the timestamp and do not send email.
  • Ideally the email should look good in raindrop -- it is likely to be the first message they see when the come to raindrop. Maybe something that is a "Welcome to Raindrop" message with an embedded youtube/vimeo URL, and in Raindrop they see it as an inline video. Andy/Bryan

• User comes back, sees metrics participation screen (disable checkbox for this alpha)
  • Add explanation that in the Alpha phase you are required to participate in the metrics offer link to account page in case you no longer want to participate.

• Inflow experience (see below)

• Ability to delete an account via Account page. James/Mark/?
  • Explain that will do our best to delete all traces of your data, however this is alpha software and there are no guarantees. Remember that from when you signed up?

Synching

• No ability to specify accounts to sync - all are done
• No ability define to define sync schedule.
• Back-end writes sync-status document with info per account. Each account will reuse/abuse http status codes (eg, 200-success, 403 forbidden, 401 no credentials.)
• Front end will invent a 'sync status line' with (eg) 'Last Synced 10 seconds ago. [Check Now] [6 new messages]
• Account preference schemas needed - this will include synch intervals, timeouts, etc (but see below - we may not allow the user to specify an exact resync period - it may depend on load.
• Implication is that there will be no "progress" resporting of a sync, just "after the event" reporting.

Futon

Need a futon which works with our auth scheme. This would be useful for diagnostics and debugging. (Eg, see logging below - log levels could be adjusted via futon instead of requiring a webby page for it.

Externals and API

run-raindrop will need to become some kind of daemon so it can be 'called' by the front-end. Couchdb externals are a 'blocking' mechanism - is not practical to serve all 200 users from a single external process.

Solution:

• Will add layer to middle-ware to route 'normal' requests to couchdb, but 'external' requests to our Python implemented processes.
• Python process will be similar conceptually to couchdb external - will take one 'blocking' request at a time on stdin/stdout. One process per user means this will scale OK. Process may self-terminate after perioud of inactivity.
• Will use OS level scheduling tool to do things like sync-messages "as often as possible" based on load. This is reason not to expose to the user things like 'resync after n seconds' Requests for things like "sync now" may also be driven via the scheduler (eg, "sync now" may simply add a sync request to the 'queue' and be done as load allows).
• Will merge 'run-raindrop' with the API; there is no need to have 2 distinct processes for this, so long as we can differentiate between 'async' requests (eg, 'start synching') versus 'sync' request (eg, API calls)
• Developers will need solutions to allow them to test code without requiring a full deployment and without checking in code; must continue to be able to test locally *before* checking in.

API Keys

We should not store API keys for APIs that Raindrop consumes in the public source repo.

• Figure out where to store API keys for hosted version.
• Work out a process for a developer to get new API keys. At the very least document on the wiki what to do, and add to it for every new API. Ideally have a web app that could help the developer get the API key and inject it into Raindrop.
• Make changes to raindrop to pull in these API keys or a web UI to allow input of the API keys.

Logging

• Need a way to expose log messages as run-raindrop.py will die.
  • Flush log messages to a couch doc every n seconds (120??) or when a WARNING/ERROR record appears, whichever happens first.

Inflow

The following Inflow items are listed in order of priority.

Compose / Reply / Forward

• Email Compose needs a button somewhere that opens up a Gmail compose window
• Twitter Compose needs an action in the Twitter view which opens a twitter compose
• Inline reply areas need to be removed for now
• Email Reply buttons need to open up a gmail reply or be removed
• Twitter Reply button needs to open up the Twitter web reply option
• Twitter RT and Star button need to be removed for now

Archive, Delete, and other actions

• Archive & Delete
  • Needs testing that they actually still work
  • Buttons should move to the upper right corner of a listed conversation
  • Conversations should be removed quickly through a slideup animate and the next conversation should appear under the mouse
• Open Button
  • Should be removed
  • Links inside the message should no longer be linkified
  • Clicking on the message text or subject should open the mail
• An "other actions" menu button can be placed at the current bottom left hand corner for holding conversation actions like "Move to Bulk" or "Mute"

Search / Find

much of this is following a ubiquity style design

• More Polished Search Entry
  • Should look like a "Starting Point" and not just an input box for search
• onFocus / onClick should open a larger popup for search,browse,and auto-complete

sketch of design

Move to bulk needs polish

• Remove the ( Not Personal ) button
  • Create a new popup/dropdown menu and add a ( Move to Bulk ) action
• We should animate the removal of a group of messages to bulk
  • Animate messages leaving Inflow
  • Ensure we don't have lots of empty white space
  • Animate a new bulk box appearing
• Add a "[x] Deliver to Bulk" option to our People / Address view

Attachments

• Clean up what we have
  • Create the following attachment type handlers
    • Video Link Attachments
      • Examples: YouTube, Vimeo
      • Thumbnail, Title, ByUser
    • Image Link Attachments
      • Examples: Flickr
        • Thumbnail, Title, ByUser
      • Example: Twitpic
        • Thumbnail
    • Expanded Link Attachments
      • Example: Bit.ly
        • Title, LongURL, ByUser
    • Image File Attachments
    • Document File Attachments
• It should be trivial for a developer to create a new backend data miner extension in a certain format so their attachments automatically show up in our default attachment type handlers
  • What needs to be done for this?
• We should be able to focus on this item when talking to developers about getting new data miners created

Security

Still mostly unknown, need to work out soon since it impacts design.

Since we need to store some passwords, then do not worry about using OAuth for twitter, just to cut down on the work, but favor OAuth in the future.

Still may use OAuth support in couch to secure the specific requests to the server. Maybe we can use the OAuth token for a "master password" for use in encryption?

Need to also make sure we show messages securely, as in strip out malicious HTML, watch out for image loads?

Operations

What pieces of software/servers are needed? Gozer to flesh out.

Random Notes

• Refresh / Send / Receive UI
• Be clear about security
  • read / write access of messages
  • how we store passwords
  • access to logs
• Get Satisfaction Integration
  • How to Leave Feedback?
  • How to lookup a parson's CouchDB
    • Show unique ID on the GS view

• What Metrics will we need?
  • Attachments
    • Types
    • Count
    • From (tweet, email)
  • Mailing Lists
  • types of list - list-unsubscribe, list-id

• Message from a New Contact

• Report a Problem Action - How to get support

Error paths

Transient errors and persistent errors.

How to know about hardware failures? What sort of monitoring is available? If doing a post-mortem how can we correct or find an issue with the hardware?

Allow a user to report a problem, but try to catch as much as we can without needing the user to report it.

Types of errors, with possible ways to report/remedy.

• Browser script problem
• Browser cannot connect to servers

• Server-to-server connection problems
  • load balancer to SSL decrypt
  • SSL decrypt to routing middleware
  • routing middleware to its config DB
  • routing middleware to the couch