Security/Features/Application Reputation/Preliminary Results

Goal

Find out if future versions of Firefox with application reputation (a new-to-Firefox safebrowsing feature using Google's safebrowsing API) improves malware detection. Using Google's API, we can never do better than Chrome. We refer to a malware download not caught by the malware check as a false negative.

Testing steps

• Enable all reputation application checks in FF 31
• Test this special build with the 657 false negative URLs from NSS labs.
• Test Chrome with the same URLs as a sanity check.

Results

• 657 false negative URLs reported by NSS Labs
• Of these, 330 URLs were not found (404, timeout, or host not found)
• Of these, 327 legitimate false negative URLs not caught by the version of FF that NSS Labs tested
• Of these, the special FF 31 build caught 155 of them, resulting in 172 false negatives.
• Of the 327 URLs that the release version of FF did not catch, Chrome caught 286 of them. The difference in coverage is due to certain metadata in the downloads (such as the redirect chain, and whether or not the user initiated the download) that we don't currently emulate.

We expect application reputation to reduce false negatives by approximately half.