Security/Reviews/Firefox/Add-on hotfix

From MozillaWiki
Jump to: navigation, search
Items to be reviewed

Introduce Feature

Goal of Feature, what is trying to be achieved (problem solved, use cases, etc)

  • Provide a low-touch way to update Firefox users in the field without a "full" software update
  • The add-on would hopefully replace the need for #.0.1 releases and we'd be able to respond to security issues even more quickly
  • Ship security and chemspill-type fixes faster (no recompile required), users download them faster (smaller download), and get them installed faster (no restart required)
  • Avoiding recompile is especially helpful given the use of PGO for binaries (PGO increases testing requirements and makes binary updates huge)
    • Mitigation for problems with app update (whether the problems affect 0.01% or 90% of users)
    • We aren't yet sure what other cases we'll use add-on hotfixes.

What solutions/approaches were considered other than the proposed solution?

  • Different update snippets / mars
  • "Special" kind of add-on
  • One-off system to download a .tgz from mozilla.org

Why was this solution chosen?

  • Minimal code changes
  • Using sec-reviewed systems and code
    • AMO
    • Add-on manager
    • Add-on system

Any security threats already considered in the design and why?=

  • Users can disable the add-on, 'perhaps delaying security fixes
  • This concern is no different than normal updates
    • Some disagree. We're always telling users "Try disabling all your add-ons" and "Disable unnecessary add-ons". I'd prefer for this addon to not be listed in about:addons.
    • Sometimes these add-ons will simply run, make some config settings, and then uninstall themselves. In the recent cases where we might have used this mechanism that would have been the way we did it.
    • How will AMO know not to give you the addon again?
  • Users who choose an add-on server other than addons.mozilla.org
    • There's a separate pref for "magic hotfix server" and "add-on update server" :)

Threat Brainstorming

When we use hotfixes to fix security holes:

  • Not all users get the update? So we'd have to ship a binary fix too.
    • We exclude this addon from the "[x] Update my add-ons" pref :)
  • If we ship a hotfix, and the next day someone downloads a binary from mozilla.org, are they vulnerable for the first few hours?
  • The extra complexity (e.g. "what happens if you install the hotfix and then the fixed version") may make it more likely to introduce regressions, including security regressions. But this is a case-by-case question, not a question about the feature.
  • With strict-compat warnings, the hotfix addon could slow down Firefox update.
    • Maybe we'll exclude the hotfix addon from compat checks.
      • Or better, happily uninstall it whenever there's a Firefox update.
  • Who will have the ability to ship these hotfixes? How does this compare to the set of people with the ability to ship new versions of Firefox?
  • By using AMO, we're adding attack surface for the 20% of users who don't have any addons installed (?)
    • Creating a custom solution is also likely to introduce flaws that AMO does not have
    • Also add-on blocklist is served from AMO
  • A silently installed/updated addon is a different risk profile than typical (even popular) add-ons. Its becomes equivalent to our existing update mechanism, so we should ensure similar mitigations are in place (i.e. code-signing with a hardcoded root)
    • Our application updates are not currently signed, though that's being added as part of the Silent Update feature
    • Our AMO updates are currently only secured by TLS
    • True but our current update service has a very small attack surface, unlike AMO. If we only rely on the integrity of AMO we have introduced the risk that compromise of 1 addon affects 100% of our users.
  • Binary addons - could complicate things, introduces code-level issues, etc - initially we are planning to ship JS, restartless addons
  • What will happen for users who have chosen "Check for updates, but let me choose whether to install them"? (Don't want to scare paranoid users away from having check-for-updates enabled.)
    • For now, those users won't get hotfixes at all :( Need UI design if we want to honor the pref and prompt for hotfixes.
    • Some users have chosen that pref because they want to know about major updates, but are fine with minor updates (including hotfixes) being fully automatic.
    • maybe trigger check/install when users do a manual app update check?

Conclusions / Action Items

{Should be completed before we ship}

  • [clegnitto] Add to the checklist for making hotfixes: ‚ÄúEach specific hotfix update would need careful scrutiny (QA, security review)
  • [mossop] Hotfix addon must be authenticated additionally by code signing cert with fingerprint hardcoded in the release.
  • [Jesse] argue more about "ask first" pref
    • find out what % have updates set to "ask first" (and currently would not get these hotfixes). If it's large may need to add UI for prompting (door hanger?), or simply ignore the "ask first" pref?
    • or treat it according to the blocklist pref?
    • or add a separate option to disable hotfixes?
    • [jesse] file a bug with [sg:want?]. the bug won't block the feature. >> bug 704985
  • [mossop] make sure setting the strict compat pref doesn't mean hotfixes prevent you from getting app updates

Background Discussion

{occured over email prior to meeting}

bsmith 2011.11.21 1554PM
Preliminary questions:
* In which scenerios will this be useful once we have the installer service + silent updates + the update
installation performance fix? In other words, what will make the new update system insufficient that 
makes us feel we need to implement this feature?
* I suspect that many of our .0.x releases would not have been averted even if we had such an addon, 
because those chemspills were in areas that are not controllable by addons. Could you please point out 
some recent .x releases that could have been avoided if we had such an addon, and some that wouldn't have 
been avoided even if we had such an addon? 
* On what timeframe would we start bundling such an addon? And, how does that timeframe compare to the 
timeframes for the various update installation improvements?
* IMO, we should move away from having addons.mozilla.org be central for the security of core Firefox, 
because we should have higher uptime requirements for AUS than AMO. For example, I think we should be 
downloading the addon blocklist from AUS instead of AMO. This would allow us to have "pull the plug on 
AMO" as a viable last-ditch solution for (security) problems on AMO. I suspect that an extension as 
proposed here would add a very strong dependency on AMO that would take "pull the plug on AMO" off the 
table for large periods of time. This seems to be moving in the wrong direction.
* Similar to the above point, I suspect some enterprises prefer to disable all access to AMO (perhaps 
at the firewall level) to decrease the odds of users attempting to install extensions. For that reason, 
I am not sure we cannot rely on connectivity to AMO for security fixes.
clegnitto 2011.11.21 1626PM
|* In which scenerios...
It gets less useful, definitely. With the add-on hotfix we could do stuff like "Only block java for people 
on es-MX who have visited [some random java site] in the past 24 hours". We have a ton more flexibility 
than we have with the updater, even if it was 100% silent. Also, the end-to end build / turnaround time 
would likely be faster (due to not having to compile across all platforms, etc), the decision to "hotfix" 
would likely be made quicker (right now we wait to get all the info because updates and builds are so 
heavyweight), and the uptake rate would be insanely faster (smaller download + installs automatically in 
the background = ~24 hours for ~100% uptake). The hotfix feature gives us all the flexibility we need and 
we can choose which is most appropriate in various situations.
Not relevant to a security context: we can do trivial a/b testing by shipping the hotfix to people on 
Aurora, randomly flipping a pref. We can ship a hotfix that pops up an info bar to Aurora testers who 
have used Java in the last 20 days and tell them to let us know if they see anything wrong with 
asynchronous plugin painting, etc.

|* I suspect that...
Yes, we have discussed this. 7.0.1 would have been prevented. We could have taken mitigation steps, 
messaged, or at least prevented the Mac Java crashes in 8.0.1. For the certificates in 6.0.1/6.0.2 we 
could have set the certs as untrusted (I think, Dan and I discussed this a bit). 5.0.1 again had java 
crashes that could have been mitigated (poor man's click to play, disable java running OOP, etc).
We even talked about the possibility of doing crazy stuff like slapping down a new updater.exe binary 
if it is broken / can't update itself, etc. We wouldn't *want* to do it and it may not be ideal, but the 
possibility is there.

|* On what timeframe...
We would not be bundling the add-on, Firefox checks without having the add-on bundled. We want the feature
to land ASAP, preferably even pushing it into Firefox 9 or 10. The development was explicitly specced out 
in a way to make engineering, security risk, and qualification minimal, and having the hotfix feature will 
take a bit of pressure off of the silent update folks.
|* IMO, we should move...
Fair enough.

|This would allow us to have "pull the plug on AMO" 
I don't think so. The feature is envisioned as a stopgap solution that is used within the 6 week window. 
The Fx updates after 6 weeks will supersede the add-on. AMO and the add-on infrastructure was chosen due 
to the fact that both have been field tested. If we need to pull the AMO plug, as long as it doesn't \
coincide with a need for a hotfix we should be fine (and can reason about the various effects). Again, 
having automatic, silent background updating should make it it so our uptake rate is amazing.

|* Similar to the above point...
I don't see how this is any different than them disabling SU checks, which many do. As long as we message 
it, I don't see a blocker here (but of course I see a concern).
bsmith 2011.11.21 1656PM
Thanks, that is very useful info.
It would be useful to do an experiment to see how we would do an extension that disabled a CA, to make 
sure we could actually rely on this mechanism.
What exactly would such an addon have done regarding the addon-related crashes? I don't see how we could 
require a 18-weeks development cycle for click-to-play normally, but then suddenly write a workable CTP 
mechanism in a few hours and ship it untested. (I know Jesse and Dan did some investigation previously, 
and discovered that our emergency fixes often caused security problems themselves.)

|We even talked about the possibility of doing crazy stuff...
Regardless of whether this is done via such an addon or via some other mechanism, I think this is a smart 
idea.

|We would not be bundling the...
How exactly would we determine which previously-not-installed addon(s) can be silently installed for this? 
Is there a special ID that we whitelist for this behavior?

|I don't see how this is...
Those sysadmins can still push out a X.0.y release to push out to all those machines.
We would have to ensure that sysadmins are aware of this mechanism and know exactly how to push it out to 
all machines so that it is active for all user profiles--including user profiles that haven't been created 
yet. Note that this is in conflict with the relatively new feature that attempts to put the user in control 
of drive-by addon installs; we would need to make an exception to allow silent drive-by install of this 
update.
More suggestions:
1. "Check for updates" should force a download of this update.
2. The presence of this update should affect the version information reported to the user in the about 
dialog box. Perhaps it should not change the version number, but there should be some notice that the 
security fix is installed.
3. Ideally, the various mechanisms we use to control addons' enablement/installation should not affect 
this addon.
4. Will this addon appear in the addon manager? IMO, it would be better if it didn't.
But, we also have to weigh the risk of making these changes against the desire to have such a feature 
early in the development cycle.
clegnitto 2011.11.21 1737PM
|It would be useful to do an experiment...
Yep, on the ol' TODO list. I actually didn't think it was possible but dan said it probably is.

|What exactly would such an addon have done regarding the addon-related crashes?
Not sure what you are referring to here so I'll address two:
* 7.0: users would have been fixed by the very act of installing the hotfix (no-op code-wise)
* 8.0: We could have turned off Java only for those on Mac OS X who had installed the most recent Apple 
java update (no crashes with the older java).We could have then looked at the data and decided if the 
problem was widespread enough to spin a 8.0.1.

|I don't see how we could require a 18-weeks development cycle...
Right, that's why I said "poor man's". Something like CTP would be a kludge and a risk in a hotfix. 
Perhaps that was a bad example, but there will definitely be cases where we need something in the short 
term and mock something up just for a 6 week cycle.
Testing this stuff is a risk for sure....but not really appropriate for a security review 

|Regardless of whether this is done ...
I agree, not sure this is the right feature for it though (also not sure what the engineering 
considerations are, there are likely many)

|How exactly would we determine which previously-not-installed addon(s) can be silently installed 
for this?
Yep, special id. Fligtar already reserved it on AMO.

|We would have to ensure that sysadmins...
Yeah, we'll let them know when it lands but again, this is a hotfix. We're trying to cover the 80% case. 
A fair amount of our user base doesn't even install the new Fx version in the 6 weeks it is current, I 
don't think we'd be making the status quo any worse off.
Also, most enterprises are on 3.6 currently, so this is a concern in the future and not so much in the 
present.

|Note that this is in conflict with the relatively...
It respects update preferences and installs into the user profile (aka seen as installed by the user). 
This was an explicit choice to minimize risk / security impact.

|1. "Check for updates" ...
Again, we're sticking as close to add-on behavior as possible to minimize engineering and testing.

|2. The presence of this ...
We thought about this and discounted it. This will generally be "silent" in that users don't need to know 
and websites won't either. It will show up in the add-ons list and there will be a pref set so even with 
a hotfix that uninstalls itself we can trace back (and prevent an install loop). It's an add-on, so if we 
later want to do this we can (set a pref, add to the UA, change the Fx version, etc). I see this more as 
policy and less critical for the Fx code.

|3. Ideally, the various...
I believe right now we key off the update prefs, though the original plan was to act like a normal add-on.
Dave and I discussed and agreed it made sense to key off the app update preferences instead (this is one 
of the only places we diverge from a regular add-on).

|4. Will this addon appear...
Yes. Again, specced out this way to reduce risk...in an ideal world it would appear in it's own category 
(l10n) and could not be uninstalled or disabled (security risk of impostor add-ons).

|But, we also have to weigh the risk...
The feature was explicitly specced out for the 70-80% case, as the original estimates were that it would 
be ~15 lines of code, not require any more extensive testing, no supporting server-side changes, no l10n, 
etc. If we had a year we could make this feature awesome, polish all the rough edges, etc. Note that 
specced out to cover the 70-80% case is different than 70-80% done.
Basically we're (ab)using the existing add-on infrastructure as a dumb pipe to shuttle bits to the majority
of our users quickly and securely. I suspect a lot of the use cases will go away when we have all the 
silent update pieces in place.
bsmith 2011.11.21 1845PM
|Testing this stuff is a risk...
I think it is very relevant because it helps determine if/when we can use this mechaism to fix security 
vulnerabilities.

|Yeah, we'll let them know when...
This, and the responses below, make me think that this wouldn't be useable as a solution for a (0-day) 
critical security vulnerability. We need an enterprise-applicable solution, and one that people can 
manually activate and verify, for security fixes.
That isn't to say that we shouldn't do it at all. Christian pointed out some good non-security-critical 
examples of where it would be useful. We just need to be clear on appropriate vs. inappropriate uses of 
this mechanism.
Also, this isn't to say we couldn't make this more appropriate as a security fix mechanism in future 
iterations, if it would be helpful to do so after silent install is improved.
Retrieved from "https://wiki.mozilla.org/index.php?title=Security/Reviews/Firefox/Add-on_hotfix&oldid=376521"