Overview

At a high level, this project aims to move the data generated and managed by Firefox OS devices to the cloud allowing users to access a synchronized version of this data at any time initially from any Firefox product and potentially from any web enabled device.

The product

The product vision is built on top of three major concepts fully aligned with Mozilla's mission.

User choice

We should offer to users the ability to decide where they want to store their data. On currently existing platforms users are tied to a specific storage: on iOS people are tied to iCloud, on Android, to Google Drive, etc. We want to give users the ability to choose the cloud storage provider they want. Ideally, we want to also be able to give them the choice to use self hosted storage like ownCloud. But this is still under discussion. Mozilla might also provide cloud storage space for users as one of these choices, but this is also still to be decided.

User privacy

In order to ensure that the data that the user sends to the cloud is protected and no one else other than the user can read it, the Firefox OS in the Cloud client solution should allow users to opt-in to encrypt the data on the client side and store it encrypted on the selected cloud storage provider. Not even Mozilla should be able to read this data or store it unencrypted. All the encryption and decryption should happen on the client side.

User identity

We want to use Firefox Accounts as the authentication mechanism for this service. Once the user links their chosen cloud storage provider credentials to her Firefox OS in the Cloud account, all that she needs to do to authenticate herself from new devices accessing her Firefox OS in the Cloud account is her Firefox Accounts credentials.

Use cases

Browser-related data

Saved passwords

Alice has been using FxDesktop for a while, and has saved the passwords for her favorite websites in FxDesktop Saved Passwords. She is already using FxSync on FxDesktop. Once she enables FxSync on her FxOS phone, the next time she visits one of these websites with her phone, the password to log in is prefilled with the one from FxSync. From now on, if she updates or adds a password on any device, this will also be updated on her other devices.

Browser History

Alice has been using FxDesktop for a while, and is already using FxSync there. Once she enables FxSync on her FxOS phone, the next time she opens the browser, she will see items from her FxDesktop history show up in the thumbnail tiles. Also, when she starts typing a URL, it autocompletes from the history that was synced from FxDesktop. From now on, if she visits a website on any device, this will also be updated in the browser history on her other devices.

Bookmarks

Alice has been using FxDesktop for a while, and has collected a nice set of bookmarks for her favorite websites there. She is already using FxSync on FxDesktop. Once she enables FxSync on her FxOS TV, the next time she browses the web with her TV, her bookmarks from FxSync are available. From now on, if she adds or updates a bookmark on any device that supports bookmarks, this will also be updated on her other devices.

Backup/Restore device

Alice purchases a new Firefox OS phone (\o/). She already owns a Firefox OS tablet and she wants to have the same experience and data in both devices. She enters her Firefox Accounts credentials while configuring her new device. Her new device installs all the applications that she has on her tablet, the homescreen wallpaper, the passcode for the lockscreen, the notification sounds. When she opens the Gallery app in her new device, she is asked if she wants to access her photo collection from her new device.

Backup/Sync contacts

So far, Firefox OS support one-shot import contacts from Gmail or from Hotmail/Outlook. We want to augment this functionality to allow one-shot import from any CardDAV server, as well as repeated imports (staying in sync), and export.

Backup/Sync media files

Bob uses his Music app to listen to music and audio files. He keeps a library with his preferred titles. He adds new songs from his desktop browser. When he uses his Firefox OS device, he can listen to these new songs if he is online. He can also choose to download them so he can play them offline.

Question: Does the proposed V3 architecture buy us anything in terms improved offline experience here? -Stephany

Alice listens to her favorite knitting podcasts -- Knitting Pipeline and The Knit Girllls -- in her web browser (while knitting beside her laptop during business travel) or via the Apple TV > Podcasts section, when she is at home. Alice can access both podcasts on the Web and in iTunes. She has a Lenovo PC so doesn't really use iTunes outside of Apple TV. When Alice sits down in her plane seat for her next business trip, plugs in her headphones and clicks the play button on the video clip embedded in the podcast's web page, Firefox Cloud asks if she'd like the video to be backed up so she can pick it up anywhere (on her phone, in case her laptop battery dies; on her laptop when it's offline during the flight with alleged but nonexistent Internet service), where she left off.

Messaging application

Alice uses her Messaging app to send and receive SMS, MMS and IM. She accesses this application from her Firefox OS tablet, her desktop browser and her Android phone. She can see and manage the history and content of the messages sent and received from any of these devices. She can continue writing an IM that she started typing on her Android phone on her desktop browser app.

File sharing

Bob wants to share a file between his desktop and his mobile phone. He accesses dummysharingservice.com, logs in with his Firefox Accounts credentials and uploads the file from his desktop. He goes to his mobile phone and logs in with the same Firefox Accounts email and downloads the file in his mobile. Now he wants to share the file with Alice. He accesses dummysharingservice.com again and uses the sharing option to send a notification about the shared file to Alice's email. Alice receives this notification and accesses the sharing service. She logs with her Firefox Accounts email and downloads the file shared by Bob.

Email

Alice has a Firefox OS device and wants to back up all of her email photo attachments, from multiple email accounts, to Dropbox and Spider Oak. She sees a suggestion for "helpful services" in the email UI, taps it, peruses services, and selects those for Dropbox and Spider Oak. Alice feels safer knowing that every photo she's ever received in an email is automatically backed up.

After a few months, Alice decides to do the same for document attachments, but only for her work email and to her organization's enterprise Box back-up, which she is required to use as an employee.

Note: Francis Djabri and Stephany Wilkes are currently working on this and presented initial designs to the UX team in early March, 2015. Please contact either of them for more information.

Other use cases

You can see the result of a previous discussion around data sync use cases here.

Data types

Looking at the proposed use cases and the nature of the data to be synchronized we can group it in:

Browser/System data

This includes things like browsing history, bookmarks, passwords, form autocomplete data, list of installed apps, system settings, etc.

In-app generated data

This includes all the data that is not owned by the system. And can be also grouped in:

Files

Things like photos, music and other media files.

Documents

Everything that fits on an indexedDB like database. Including in-app specific settings or things like SMS, call history, alarms, etc.

Projects

The Firefox OS Data Sync team will be working on several projects during 2016. See Firefox OS Cloud Roadmap for an extensive list of product ideas.

Firefox Sync

We have implemented import-only bookmarks and history synchronization for the Smart TV in FxOS 2.5. We will be enabling FxSync on master for phone foxfooders soon.

The next logical step is to finish the work that we started with Firefox Sync in a way that Firefox OS can catch up with other Firefox Sync clients (Desktop, Android and iOS). So we need to work on exporting data from the collections that we already have implemented (bookmarks and history) and work on importing/exporting from the collections that it still doesn't have (passwords, add-ons, tabs, preferences, forms, clients). Some of these may not make sense on Firefox OS though.

Also for bookmarks, we still need to define and implement an UI to display the synchronized bookmarks on the phone.

Finally, we need to allow the creation of new Sync users from Firefox OS. This involves generating the 'crypto/keys' object containing a new bulk key bundle, encrypt it with 'kB' on the client and uploading it to the server.

Stage 1 (completed): Read-only history and bookmarks sync on the Smart TV.
Stage 2 (completed): Read-only history sync on the phone.
Stage 3 (to be discussed, see ML): Read-write history sync and passwords sync on the phone.
Stage 4 (to be discussed, see ML): Sync the rest of the collections as and when the phone starts supporting these features.

Media Files Sync

Planned for 2016. FxOS already allows users to store and view media files using storage that is connected to the device (for instance, an SD-card). Mounting remote cloud storage space (for instance the user's Dropbox or ownCloud account) as a (cached) virtual storage device makes it possible to browse the files on there 'as if' they were on a local storage device.

Contacts Sync

See this mailing list thread for work going on during 2015-Q4. During 2016, we would like to improve the existing sync (currently more a one-shot import) with Gmail and Outlook contacts, and expand it to allow sync with any CardDAV server.

Backup / Restore

Planned for 2016. It should be possible to take snapshots of the entire data state of a Firefox OS device, and later restore that snapshot onto the same or a new device. During Mozlando we will discuss whether this can be a pure peer-to-peer architecture (e.g. using RTCPeerConnection), or whether it will be necessary to store these snapshots on a server where the user has storage space. See https://wiki.mozilla.org/Firefox_OS_Cloud_Backup_Restore for an extensive list of product requirements.

Data Sync for the Web

Apart from launching specific sync features on upcoming Firefox OS versions, we will aim to discuss, build, and standardize the general architecture of data sync technologies that are currently missing on the web platform, but are needed to make data sync on the (mobile) web as easy to use as the data sync features of proprietary single-vendor platforms.

The solution(s)

For Browser/System data

For Browser/System specific data we chose to use the existing Firefox Sync platform. That will give users a way to backup some of the browser-related system data like saved passwords, the browser history, their bookmarks, etc. and to access this data from any Firefox product (Desktop, Android, iOS, OS).

For in-app generated data

We are still not sure about which approach we should take to allow the apps to store and synchronize in-app generated data with a remote storage. We know some requirements we would like to address though.

Requirements for in-app data sync

Authentication

Firefox Accounts should be the authentication mechanism used for this. Firefox Accounts is also the authentication mechanism currently used by Firefox Sync, so it makes a lot of sense to have the same for in-app generated data. Using Firefox Accounts would enable us to do the data encryption in the client without worrying about storing any private key or secret in the clients. We can obtain a key derived from the user's Firefox Accounts and encrypt the local data on the fly before sending it to the cloud. Assuming that we finally allow 3rd party storage providers, the authentication keys for these providers selected by the user could be stored in a Mozilla server also encrypted with a symmetric key that the client will provide on every sync request. That way if the Mozilla server is compromised, the attacker won't get access to the remote storage accounts.

Files and documents

The solution that we provide to developers should allow them to store and synchronize files and documents, understanding files as data stored "directly" on the file system (i.e. via DeviceStorage or even the File System APIs and documents as the data stored in a local database most likely an IndexedDB one.

It should be cross browser.

Whichever solution we find for this, it should work in all browsers. If we end up creating a JS library, we need to be sure that it works (or will work, thinking about IndexedDB support) in all the major browsers. We believe that we can avoid adding a new web API for this, but in case that we finally need to do it, it would be great if we could standardize it and avoid repeating the DataStore fiasco.

Offline first.

Ideally we should provide a solution where consumers can store the data locally first and then specify that the data needs to be synchronized with a given remote endpoint. The app developer shouldn't need to know the underlying details of the sync protocol that we choose to implement. We should provide them with an abstraction to request this synchronization in the form of a JS library.

Avoid enforcing another client storage solution.

We already have enough storage APIs in the client side at least on Firefox OS to add another one. We don't think it is a good idea to force developers to use another API for remote data synchronization. Instead, we would prefer a solution that allows developers to keep using the API of their current data source (let's say IndexedDB) for adding, editing and removing records and performing searches. While we can provide an easier abstraction of the current APIs (IndexedDB is known for being a powerful but complicated API), we should still allow developers to keep accessing and using them.

Avoid data duplication.

One of the issues that we currently have with DataStore is that we potentially create several copies of the same data across Firefox OS. Ideally, we should not require to do the same with the solution that we choose for the client side of this system. So we should either have a solution that allow us to keep using the same data sources that we use currently but somehow adding the sync capabilities to it or a way to migrate the existing data to a new data source with already sync enabled capabilities. We will certainly need to add meta information along with the existing data that we have in our apps, but this data should have an unique source.

Proposals for in-app data

Proposal 1: Generic storage and storage provider proxy

The Cloud Services team has been working on a generic storage service to allow 3rd party apps to store and synchronize arbitrary data, attached to a Firefox account. They've also supported the idea of having an intermediate service (could be part of the previous one) that would act as a proxy for different storage providers (one of them could be the Mozilla generic storage service mentioned before).

With that kind of services we could have a high level architecture like the following:

The code and documentation for this generic storage can be found at:

Proposal 2: Universal Storage API and Virtual Storage Interface

The Taipei team is currently working on a proposal to extend the DeviceStorage API to allow remote storages. You can read more about this here

Specific protocols

For some cases like the Email (IMAP, POP3) or Calendar (CalDav) apps we are already implementing specific sync protocols. We need to keep supporting and adding more of these specific sync protocols. For now the list of identified ones are:

Gmail for Contacts. We are currently importing contacts only.
Outlook for Contacts. We are currently importing contacts only.
CardDav for Contacts. One of the most wanted features (269 votes so far). Check bug 859306.

Ideas for Architecture Improvements

SyncEngine shared library

Right now, the sync engine lives only inside the Sync app that we added for synchronizing bookmarks and history. This works fine as long as the data source to be synchronized is accessible outside of the owner app (via DataStore for example). But it makes it impossible to synchronize an app's data that is not exposed outside of the owner app. Also, having the Sync app managing the synchronization of any kind of data doesn't really scale very well and causes potentially unnecessary data duplication. It makes more sense that each app handles the synchronization of its own data. So in order to allow the addition of new collections (not only existing Firefox Sync collections, but also any other type of collection like contacts, call log or settings data) from any application we need to package the logic of the sync engine in the form of a shared library that can be used by any Gaia application.

Sync collection extension mechanism

Apart from creating this shared library to ease the synchronization process from other apps, we also want to create a way for apps to register themselves in the Firefox Sync machinery. This is, to be able to add their collections to be listed within the Settings UI and to register themselves to be waken up periodically by the sync scheduler that lives in the System app.

Push notifications in Kinto

Because we want to avoid unnecessary battery and network consumption by polling constantly from the device but we still want the data to be synchronized as soon as possible, we want to add push notifications to Kinto. This way, when something changes in a remote collection, the server will send every client of this collection a notification about this change so it can update itself accordingly.

Firefox Accounts improvements on Firefox OS

This is not entirely part of sync, but it is quite related. We are using Firefox Accounts as our identity mechanism. Firefox OS the only platform that still uses BrowserId assertions while other platforms are using the OAuth flow instead. This forces Relying Parties that want to benefit from the SSO system that we have on FxOS to use the non-standard mozId API. It also forces every server side part that we want to create for syncing to support both BrowserId assertions (if we want to sync from FxOS) and OAuth tokens (if we want to sync from Desktop). We aim to move to OAuth as soon as possible.

Related work

Toolkit team

Mailing list

mozilla.dev.fxos.sync

Bugs

Full Query

ID	Summary	Component	Priority	Resolution	Depends on	Blocks	Whiteboard	Blocking b2g	Feature b2g	Target milestone
824026	[meta] Firefox Sync client for Firefox OS	Sync	P1	WONTFIX	1212023, 877648, 1168153, 1168160, 1168164, 1168171, 1168185, 1174191, 1180330, 1182001, 1183103, 1187387, 1187411, 1191770, 1191771, 1191773, 1191774, 1191776, 1194091, 1194092, 1194096, 1194097, 1194098, 1194104, 1194108, 1195647, 1196096, 1196236, 1202347, 1202382, 1202383, 1202627, 1203898, 1205220, 1205239, 1205292, 1205331, 1205901, 1205933, 1206009, 1206010, 1206012, 1206014, 1206695, 1207483, 1207488, 1207521, 1207654, 1207725, 1208352, 1208362, 1209906, 1209934, 1210006, 1210356, 1210412, 1210442, 1210473, 1210480, 1210697, 1210698, 1210725, 1210732, 1210858, 1210902, 1211367, 1211370, 1211376, 1211401, 1211469, 1211508, 1211537, 1211555, 1211606, 1211767, 1211833, 1212187, 1212394, 1212716, 1212776, 1212777, 1212874, 1212894, 1212997, 1213239, 1213249, 1213362, 1214105, 1214193, 1214496, 1214570, 1214661, 1214979, 1215169, 1215427, 1215436, 1215458, 1215459, 1215463, 1215473, 1215509, 1216022, 1216163, 1216180, 1216451, 1216456, 1216591, 1216614, 1216616, 1216645, 1216854, 1216855, 1217331, 1217340, 1217349, 1217352, 1217376, 1217380, 1217381, 1217385, 1217463, 1217760, 1218278, 1218286, 1218288, 1218293, 1218303, 1218314, 1218383, 1218420, 1218432, 1218440, 1218441, 1218680, 1218687, 1218695, 1218701, 1218724, 1219108, 1219151, 1219160, 1219162, 1219245, 1219259, 1219275, 1219610, 1219621, 1219708, 1219709, 1220089, 1220122, 1220528, 1220541, 1220573, 1220600, 1220605, 1220961, 1220996, 1221226, 1221227, 1221234, 1221236, 1221387, 1221420, 1221427, 1221472, 1222016, 1222359, 1222397, 1223379, 1223417, 1223781, 1223812, 1224183, 1224194, 1224198, 1224203, 1224204, 1224205, 1224519, 1225017, 1225046, 1225450, 1225700, 1226453, 1227081, 1227084, 1227085, 1227429, 1227593, 1228964, 1228966, 1228967, 1229023, 1229920, 1230142, 1230210, 1230224, 1230521, 1230588, 1231929, 1231933, 1232276, 1233355, 1233379, 1234172, 1235326, 1235328, 1235330, 1235343, 1236499, 1236502, 1236863, 1237568, 1237570, 1237581, 1237647, 1237651, 1237731, 1238505, 1239266, 1239546, 1241928, 1242346, 1242394, 1242439, 1242918, 1244167, 1244168, 1244169, 1244170, 1244174, 1250834, 1251153, 1256227, 1260089, 1274457	1034937, 1161657	c=browser u=user	---	---	---
1227437	[meta][Data Sync] Contacts synchronization	Sync	--	WONTFIX	859306, 1227595	1161657		---	---	---
1227438	[meta][Data Sync] Media files synchronization	Sync	--	WONTFIX	1035053, 1164750, 1227596	1161657		---	---	---
1227441	[meta][Data Sync] Backup & Restore	Sync	--	WONTFIX	1227598	1161657		---	---	---

Firefox OS Data Sync

Contents