Thunderbird:Pluggable Store Migration
Migration is conceptually fairly easy. We iterate over the folders in the source store, create the folder in the target store, and then stream each message in the source folder to target folder. In order to avoid file collisions, especially for file based stores like berkeley mailbox and maildir, we'll need to write to a different directory and then either point the account at the new directory, or delete the original folders from the directory. But we also have to worry about other files stored in the profile directory (msgFilterRules.dat, popstate.dat, feeds.rdf, hostinfo.dat and the like). Either we hardcode the knowledge of those files, or we clone the whole directory, delete the folders from the clone, and then stream the folders into the clone, and then change the server to point at the new clone directory. Or temporarily point the server at the clone directory, delete the folders from the original, and stream from the clone to the original, and then point the server back at the original directory. I want to be very cautious about deleting the original store, since for the pop3 case, that's the user's mail. But we have to give the user some way of deleting the original store. And I want to make unwinding from errors in the migration easier. That all points to a strategy of leaving the original server directory alone, and writing into a new directory, and then switching the server directory at the very end. But some users are very particular about their server directory, so changing it is potentially rude. We could stream to a new temp dir, rename the old dir, rename the new temp dir to the original dir name, and then delete the renamed old dir.
What if the store being migrated is a cloud store? In that case, the special files like popstate.dat are probably the only ones we care about copying.
I think we need to create a temporary dest server object and set its directory to the target directory, set the pluggable store for the temp dest server to be the new store type, and then create a root folder for that server. That root folder will be used as the root for all the folders we create after that. We'll need to remove the temp server when we're done.
We probably want to avoid as many notifications as possible - we won't want gloda to start indexing these new folders and messages, for example. And we don't want removing the temp server to cause any issues (like deleting the storage associated with the server). I'd like to write the migration code in js, but that could make doing any low level trickery impossible. Perhaps we could turn off folder notifications, but other stuff might be going on during migration. Do we want to force migration to run before startup finishes?
We'll need to preflight the storage requirements for the migration - probably requiring double the storage used by the current server directory would be a conservative estimate.
What, if anything, should we do with .mozmsgs directories? I guess we don't want to copy them because we'd need to mark the headers in the .msf file in the new directory as being indexed. The maildir store is special because the indexers should be able to use the messages in the store directly, and not make a copy. We need a way for the os search indexers to know this. Perhaps a boolean attribute on the store. Will the indexer need to know how to get the file from the message id? More likely, we'll need to be able to go from the file name to the message id, in order to open search hits. Perhaps we can add a new interface for this, and if the pluggable store implements that interface, the indexer will use it.
An even simpler route for local accounts would be to simply create a new account/server with the desired storage type, and move all the folders from the source server to the target server, and then tweak the source server to use the new server's directory and storage type. But we'd still have the issue of indexers doing the right thing. Since migrating storage is going to change message keys, we may need to re-index. I think we actually want to do a folder copy, both because we only allow copies across servers, and calling it a move is technically wrong because the folder uri won't change. Asuth has suggested sending a compact complete notification to gloda, which tells it the message keys have changed, which it can handle.
What about offline stores? In theory, we could just let nature take its course and let autosync repopulate the offline store, but that harm bandwidth-limited folks. I think we want to do something slightly different. We don't want to create a folder per se, just an offline store. And we only want to stream messages that we have offline. I think we want to call nsIMsgFolder::GetOfflineStoreOutputStream() and stream the message to the offline store. So the algorithm for offline stores would probably look something like this, for each folder:
For each header in the source db,
copy hdr to the dest db if msg is offline, stream the message to the offline store
I could either add a method to nsIMsgFolderNotificationService that tells it to suppress notifications for a particular server, or add a notifyItemEvent notification to notify listeners to ignore notifications for a server. The former seems more direct.
We'll want to be able to give progress info - this should be done by summing up the sizes of all the messages in all the folders, and then giving progress as a percentage of the total size we've migrated so far.
We could create an nsIMsgIncomingServer object in js, that just supports the minimal methods needed by the pluggable store, i.e., getting the root folder, and perhaps a couple other things. But we'd probably need to be able to create a real folder object of the right type (e.g., nsLocalMailFolder), though maybe not. In theory, this could be a way of avoiding having notifications sent out.
We don't want new mail to get added to an account whose store we are migrating. We could lock the folders, or we could go offline, or disable checks for new mail, or some combination of the above. We'd also want to prevent the user from moving messages while migration is going on.
