Pancake Infrastructure
See the diagrams attached to https://bugzilla.mozilla.org/show_bug.cgi?id=731228
Contents
Web Server
fxhome-web
What does it do
The fxhome-web module contains the main pancake web application.
It is a Pyramid web application that is mounted on http://pancake/*
The web application is split up in several smaller web applications that are all 'glued' together by NGINX. The other services like pancake-social and pancake-search are mounted on top of this application in NGINX.
What does it store
It stores Pyramid/Beaker session data in the MemBase Server. Sessions currently only contain the email address of the logged in user.
What does it talk to
It talks to BrowserID to log users in.
It talks to the Pancake MemBase server to store/load session data
It talks to the Pancake User API Server to store and retreive user records.
It talks to the Pancake Thumbnailer API to generate thumbnails for sites when showing result 'tiles'.
fxhome-lattice
What does it do
Lattice is the service that builds graphs from a user's browsing data. It receives commands from the iOS application to records visited sites, links and organizes this data in a graph. Lattice is implemented as an HTTP REST API.
The API is documented at https://docs.google.com/spreadsheet/ccc?key=0AgzuG8vbAx63dHYwUmFBNThBbk8tdmdDTmtaVUZUaUE
It is a tornado web application that is mounted on http://pancake/lattice/*
What does it store
The lattice service stores all its data in the Neo4J Server. The data that it stores is described in the Neo4J section.
It also stores the same data in the Elastic Search server.
What does it talk to
It talks to the MemBase server to retrieve a user's session object. To find out if the request is authorized.
It talks to the Neo4J server to store and retrieve the user's browsing history in a graph based format.
It talks to Elastic Search to do search and generate 'top sites' for a user.
It talks to the Pancake Thumnail API when it returns a response with a list of sites.
pancake-social
What does it do
The pancake-social module implements the 'social' code. It is responsible for talking to Twitter and Facebook to collect the most recent messages from a user's message stream or wall.
It is a Tornado web app that is mounted on http://pancake/social_api/
What does it store
It does not store anything.
What does it talk to
It talks to the MemBase server to retrieve a user's session object
It talks to the pancake-user-api service to retrieve the user record which includes the Twitter and Facebook OAuth tokens.
pancake-search
What does it do
The pancake-search module implements the search code. It is responsible for talking to Bing and Twitter to run searches.
Searches are fully anonymous, no user info is currently passed on to Bing or Twitter.
It is a Tornado web app that is mounted on http://pancake/search/*
What does it store
It does not store any data.
What does it talk to
It talks to the Bing Search API
It talks to the Twitter Search API
User API Server
pancake-user-api
This implements our user database. It is a tiny HTTP REST API wrapped around a SQLite datbase.
It is not public facing and just runs internally for other Pancake components.
What does it store
It stores users and user_properties in a local SQLite database:
create table users ( username text primary key, created integer not null )
create table user_properties ( username text not null references users on delete cascade, property_name text not null, property_value text not null )
The following properties are currently used:
- facebook_access_token
- twitter_access_key
- twitter_access_secret
- status (for beta signup, set to 'beta')
The Facebook and Twitter properties are storing OAuth tokens.
What does it talk to
It does not talk to anything.
Pancake MemBase Server
Membase
What does it do
Store temporary and semi-permanent data used by the Pancake components. Currently only fxhome-web stores session data in MemBase. Other compoentns are expected to cache data in MemBase where appropriate.
What does it store
Pancake web application session data. Session objects only contain the user's email address.
What does it talk to
Nothing.
Pancake Neo4J Server
Neo4J 1.6.0
Pancake Elastic Search Server
What does it do
Neo4J is a graph database that stores the data that is managed by the fxhome-lattice module.
What does it store
It stores the user's browsing data / history in a graph structure.
The nodes of the graph are sites that have been visited and contain the following information:
- The date of the visit
- The URL of the page that was visited
- The title of the page that was visited
- The id of the 'stack' that this visit is part of
- The id of the 'session' that this visit is part of
- A unique ID of the device that generated the visit
What does it talk to
It does not talk to anything. It only exposes an API to the fxhome-lattice service.
Pancake Elastic Search Server
Elastic Search 0.19
What does it do
It stores a full text index of the user's browsing data. It is pretty much a copy of the data in Neo4J except structured in such a way that we can fulltext search on it.
What does it store
The same data that is stored in Neo4J.
What does it talk to
It does not talk to anything. It only exposes the Elastic Search API.