Raindrop/Install

From MozillaWiki
Jump to: navigation, search

Raindrop

Check out the Hacking page for more context on the code.

Late Breaking News/Known Problems

  • Integration with twisted logging is poor.
  • We are in the middle of moving from using the manually executed 'run-raindrop.py' script to the couchdb managed 'couch-raindrop.py' script which is controlled by the front end. If both processes are running you can expect one to see regular conflict errors.

Installing Raindrop

couchdb

Install couchdb, version 0.10 or later. 0.9 will not work. See:

When building couchdb from source, it can be run without installing via a 'dev' install: 

cd couchdb; ./bootstrap && ./configure && make && make dev
./utils/run

raindrop

  • Install Python 2.5 or 2.6 (note that Python 3 is not yet supported).
% hg clone -r 0.1 http://hg.mozilla.org/labs/raindrop

The trunk code is very much in flux, but if you prefer to use it, leave off the tag: 

% hg clone http://hg.mozilla.org/labs/raindrop
  • Execute the 'check-raindrop.py' script in the raindrop/server/python directory.

This should check all dependencies and let you know what you are missing, then check your couchdb for certain configuration requirements.

If you are brave, try executing 'check-raindrop.py --configure' (prefixing that command with 'sudo ' on some platforms). This will attempt to automatically install all dependencies necessary and configure couchdb.

If you are less brave, run it without the --configure option, then manually resolve any dependency or configuration issues it reports, then repeat until no problems are found.

A summary of the dependencies are: setuptools, twisted, PyOpenSSL, Skype4Py, python-twitter, simplejson, feedparser and paisley.

Notes

Notes for mac users:

  1. ERROR: module 'twisted>=8.2' has version conflict: (Twisted 2.4.0 (/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python), Requirement.parse('twisted>=8.2')) happens when trying to install twisted to the system python. Install Python 2.6 and make sure you're using it.
  2. doing easy_install PyOpenSSL after May 1st, may install some badly. It may be a PyOppenSSL 0.9 issue. 0.8 is known to work. Try the latest code, but if that does not work, switch to 0.8.

Twisted 0.9 (which is what you get if using easy_install or check-raindrop.py --configure) seems to cause problems with raindrop: [1].

Running Raindrop

After check-raindrop.py reports everything is in order, the next step is to set up the accounts to use with raindrop, then have raindrop fetch the messages

Configuring

Raindrop uses a file called .raindrop to hold your account configurations. This file lives in your home directory.

Important: the user account that runs CouchDB will need to have read and write access to the .raindrop and .raindrop.temp file in your home directory for it to run.

If you installed CouchDB with the standard defaults and you just want to configure one Gmail and one Twitter account, you do not need to manually set up the .raindrop file, and you can skip this section. You will be prompted in the web UI to set up the Gmail and Twitter account via OAuth. However, the user account that runs CouchDB needs to have read and write permissions for the .raindrop and a .raindrop.temp file to work.

Manual .raindrop configuration

If you just want to These steps should become less necessary as time moves forward (it will be possible in the future to do this work though a web interface), but for now, you need to set up the accounts you want to use in raindrop via a .raindrop file:

configure raindrop by editing ~/.raindrop

if your 'local' couchdb isn't on localhost, add a section along the lines of: 

[couch-local]
host=hostname
port=port

Add one new section per account; note the "account-" prefix is important when defining accounts:

  • Add gmail accounts along the lines of
[account-gmail-username]
proto=imap
kind=gmail
username=username@gmail.com
password=topsecret
ssl=True
  • Add twitter accounts along the lines of
[account-twitter-username]
proto=twitter
kind=twitter
username=username
password=topsecret
  • Add imap accounts along the lines of
[account-imap-username]
proto=imap
kind=imap
host=imap.example.com
port=993
username=username
password=topsecret
ssl=True
  • Add skype accounts along the lines of
[account-skype-username]
proto=skype
kind=skype
username=username
password=topsecret
  • Add RSS feed along the lines of
[account-rss-feed]
proto=rss
uri=http://example.org/rss.xml

Accounts name must be unique. A 'rss' account already exists, which fetch messages from getsatisfaction.com

First Run

If you did not manually configure the .raindrop file, then you can run this command in the raindrop/server/python directory to set up the web UI: 

% ./run-raindrop.py

When that is done, go to this URL:

http://127.0.0.1:5984/raindrop/inflow/index.html

You will be prompted to set up a Gmail and Twitter account via OAuth. Finish the account setup flow. Once you are done you will have to do another command to start syncing messages:

Once you are done with the web account setup or manually configuring ~/.raindrop, then go to the command line, to the raindrop/server/python directory and run the following command: 

% ./run-raindrop.py sync-messages --max-age=5days

It will take a few minutes for run-raindrop to complete syncing messages. The above command gets the last 5 days of messages for each account configured above. Once the above command finishes, then open your favorite Firefox or WebKit-based browser and go to:

http://127.0.0.1:5984/raindrop/inflow/index.html

You should see some of your messages displayed.

There are many issues with the UI and how messages are processed, but hopefully you now have a feel for how raindrop works.

Controlling

The run-raindrop.py script is capable of fetching messages, re-running extensions, etc. It is expected that over time, this functionality will be rolled into the couchdb controlled couch-raindrop.py script. As a result, some functionality is duplicated in these scripts.

In particular, there is no protection against 'couch-raindrop.py' and run-raindrop.py attempting to perform the same thing at the same time, which will cause many errors to be logged. Before using the advanced features of run-raindrop, you should ensure couch-raindrop.py is not running by executing a command similar to: 

% curl http://127.0.0.1:5984/raindrop/_raindrop/exit

When run-raindrop.py is executed, it always checks to see if the accounts and other documents required by raindrop (eg, the front-end applications) are up to date, then performs the requested operation, then terminates. If no operation is requested a web-browser pointing at raindrop will be opened. This means you can edit any of the files in the 'schema', 'client' or 'couch_doc' directories, then see those changes reflected in the front-end when run-raindrop is next started. We don't detect the changes while we are running though, only at startup.

Get messages from all accounts:

% ./run-raindrop.py sync-messages --max-age=2days --folder=inbox

Both --max-age and --folder are optional - see --help for details. Reload the front-end applications to see the messages as they arrive.

Unprocessing:

To delete all the intermediate messages in the DB, execute: 

% ./run-raindrop.py unprocess

The next time you execute a 'process-backlog' command, all your messages will be reprocessed.

Error handling:

If there is an error during a 'process' operation (ie, an exception in the converter), we will write an 'error document' and continue processing the work queue. We don't make a very loud noise at the end when this happens - you need to notice it in the log output as it is running. At any time you can execute: 

% ./run-raindrop.py retry-errors

To re-process those messages - but this will generally result in exactly the same error unless the converter has been fixed to deal with the condition causing the error. Alternatively, execute 'unprocess' to force reprocessing of all messages, not only those which previously caused an error.

Testing

Test everything using the test suite 

% ./test-raindrop.py
Retrieved from "https://wiki.mozilla.org/index.php?title=Raindrop/Install&oldid=220931"