Add-ons/Reviewers/Guide/Reviewing
Contents
Technical Review
Introduction
Add-on reviewers help ensure add-ons are safe to use, reliable and clearly presented to users. We also provide quick, clear, and actionable feedback to developers if issues are found with their add-ons.
All decisions should be based on the official Review Policy, please make sure you have read and understood the policy. If you have any questions or need clarifications, the admin team is happy to help. There are no dumb questions when it comes to the review policies!
The add-on review process consists of the following phases:
- Automatic Review: When an add-on is uploaded, it undergoes a number of automatic validation steps for the general safety of the add-on.
- Content Review: Within a fairly short time after submission, add-ons are inspected by a human to ensure that the listing adheres to content review guidelines. This includes metadata such as the add-on name and description.
- Technical Code Review: The source code of the add-on is inspected to ensure it is in compliance with our review policies.
- Basic Functionality Testing: Once the source code is verified safe, the add-on must be given a basic test in functionality to ensure that it acts as described.
In the following sections, you will learn more about each phase of review and how to go about it. Before we go into detail about the phases, we’d like to tell you a few helpful tips on how to get your system ready for reviewing.
Setting up a Review Environment
You can complete most of the review using the reviewer tools at addons.mozilla.org. A file viewer is available that will also diff between add-on versions, and the validation report shows additional information that may be helpful during the review. The validation results are also shown inline in the file viewer.
Most of the reviewer tools pages are fairly straightforward, but we have prepared a video that will give you an introduction to the tools. We recommend taking a moment to watch it now.
Security Considerations
As a reviewer, you have access to systems that allow approving and rejecting large amounts of add-ons. Add-ons that have not been reviewed may contain code that aims to control your computer or steal personal information, such as login tokens to addons.mozilla.org.
As such, you need to be conscious and security-aware when accessing and using the reviewer tools. Please adhere to the following security guidelines:
- When accessing the reviewer tools, never do so in the same profile that you are testing an add-on with. It is recommended to use a separate profile only for accessing the reviewer tools that has third party add-ons disabled.
- Use a separate profile for testing add-ons, which you can regularly throw away and recreate.
- When testing add-ons that require additional software (e.g. through native messaging), always make use of a VM to facilitate testing. You can use VirtualBox along with a free image from modern.ie. Make use of snapshots to reset the VM after installing third party software.
- Never log in to the reviewer tools from within the VM where you install third party software related to add-ons, even in a separate profile.
If you become aware of a potential leak or misuse of your AMO credentials, please get in touch with the admin team at amo-admins [at] mozilla [dot] com immediately so we can analyze ensure your account was not accessed by a malicious third party.
Automatic Review and Risk Categorization
When a new add-on version is uploaded to addons.mozilla.org (AMO), the addons-linter runs a set of validation checks for potential issues. Following this validation, add-ons are signed and listed on AMO immediately. They are then shown in the list of auto-approved add-ons in the reviewer tools.
As you can see from the screenshot above, the review queue is sorted according to weight, otherwise known as risk. The highest weighted add-ons are sorted at the top of the list. A number of factors are included in the risk calculation, including abuse reports, therefore add-ons that turn out to be malicious will quickly bubble up in the list.
While there are a lot of add-ons in the auto-approved list, we ask you to focus on those with the highest calculated risk. These are the most important add-ons to review.
Content Review
When an add-on is submitted and passes automatic review, a team of content reviewers take a look to ensure the add-on listing is in line with our content policies. This is useful for example to filter out spam submissions or obvious malicious content.
As a technical reviewer, you must be familiar with the content review guidelines and be ready to enforce them. Add-ons may be rejected in content review for the following reasons:
- Obscene or pornographic images in the icon or screenshots
- Hate speech in the add-on listing (note: anything promoting Nazism or that uses Nazi symbols must be rejected)
- The add-on is spam
- The add-on infringes on Mozilla copyright or trademarks.
Content reviewers make decisions solely based on information provided in the add-on listing. It is possible that an add-on’s metadata could be considered acceptable but the code is unacceptable, and vice versa.
Confirming approval as a technical reviewer also marks the add-on as approved for content review. If you see an add-on that has not been content reviewed, please undergo a full content review of the metadata.
Now is a good time to take a minute and read the content review guidelines and the linked Mozilla Acceptable Use policy to make sure you have understood them.
Technical Code Review
This is where you will be spending the most amount of time. With help of the review tools, you will be able to determine if the code provided by the developer complies to our policies. Along with actually viewing the code, you should make yourself familiar with the review history to ensure consistent replies.
In this section we will introduce you to the review history, explain how to go about the code review, and end with a few tips and tricks to help you in reviewing. To get started, click on an extension in the queue to be taken to its review page.
Review History
When looking at the review page, you will see information on the latest add-on version that was submitted. It may contain notes from the developer in the “Notes to Reviewers” section. There may also be past conversation between reviewers and developers that you should read.
Unless an add-on was just recently submitted, it will have past versions that will also contain notes. You do not need to read through each and every version’s notes, but it is helpful to read up on the notes from the last rejection or confirmed approval to ensure all issues are taken care of.
Once you have obtained an overview of the review history, you can go on to review the add-on’s code.
Code Review
Add-ons that have not previously been reviewed by a human need to be inspected in their entirety. Once a manual review has been completed successfully, it is sufficient to review the changes since the last confirmed approval or rejection. If the review version contains a “Compare” link, you can use this to compare against the last confirmed version. Otherwise, use the “Content” link to view the add-on source code.
The file viewer commonly opens with the manifest.json, which allows you to obtain an overview of the add-on’s functionality. You should now begin to review all the files in the add-on, starting from the first file at the top, subsequently going through each file until the last one is reached.
When finding issues during a review, note them in the review comment field on the review page, preferably using the canned responses. You can read more about how to best make use of the responses and find examples that will help you make better review decisions in our Review Decision Guidelines.
It is very important to include the file and line number of each issue you find. This will help developers quickly identify and address issues. While the issue may seem obvious to you, it may not be readily apparent to the developer. Including line numbers quickly resolves this issue and reduces frustration on all sides.
If the same issue occurs more than a few times in each file, you should cite at least three lines where the issue occurs. If there are more than three occurrences of the same issue, you can let the developer know that similar issues exist in other parts of the file.
Libraries and other Minified Code
It's very common for add-ons to use libraries or frameworks such as jQuery or React. Some libraries however, were written with only websites in mind, where security boundaries are very strict. Therefore, we have to review libraries, keeping in mind they are being used in a more privileged context.
Our validation checks will attempt to detect known third-party libraries, but won't work in every case. If detected, the library’s code won't generate warnings and it will be greyed out in the code viewer and does not have to be reviewed.
Other libraries have to be reviewed in a similar way that normal add-on code is reviewed. Please go through the library code and make sure it complies to our policies. If you come across library methods that are potentially unsafe, find out if the method is actually used in the add-on. For example, jQuery uses the $.parseHTML() function which has security considerations, but we still allow the use of jQuery if the add-on does not use these functions.
The developer must provide links to the library files they used in accordance with our library usage guidelines. If you come across a minified library that is not known to our system, please take the following steps:
- Check if the developer has provided a link to the exact file being used, as per our library usage guidelines.
- If the links are not available, please request more information using the reviewer reply feature.
- Verify the link is from the original maintainer’s website
- Verify the file included in the add-on is an exact match to the linked original file on the maintainer’s website.
- Search for the corresponding unminified file on the maintainer’s website
- Review the unminified file as described above
If the links are not available or do not point to the original maintainer’s website, please request more information from the developer using the canned response. If the file contents do not match, you can reject the add-on. Canned responses are available for this purpose. If you believe a library is used sufficiently often that it should be added to our automatic checks, please get in touch with the admin team.
Aside from libraries, many add-ons include minified, obfuscated or otherwise machine-generated code. Since this code can't be easily reviewed without the original sources, only admin reviewers should review them. The Add-on History entry should indicate if the source code has been provided by the developer. If that's not the case, use the canned info request about minified code. You can read more about our Source Code Submission guidelines.
Tips & Tricks
Reviewing add-ons is a lot about following data around through the security boundaries within the add-on. A web page has less privileges than a WebExtension content page, which has less privileges than the WebExtension background page.
We recommend that you concentrate on finding code where data is being injected or executed (e.g. use of innerHTML), then backtracking to see where the data originates to determine if it is safe.
Likewise, when data leaves the user’s computer, you’d want to backtrack to the origin to identify what exact data is being transmitted.
If you stop at each function that has a potentially malicious nature and then track forward to see where it is used, you will end up reviewing many files more than once. This approach takes substantially more time, and we do not recommend it.
Basic Functionality Testing
The last step in a review is to install and test the add-on. You can do so by navigating to `about:debugging` in a separate profile and installing the add-on as a temporary add-on. If you have downloaded and extracted the add-on you can also make use of web-ext run to start a browser with the add-on installed.
For the rare circumstance that you need to test an add-on persisting over a restart, you will need to use Firefox Nightly with signing disabled or an unbranded build to test.
You are not expected to extensively test each feature of the add-on. Instead, expect to spend not more than 10 minutes reviewing the add-on’s functionality to see if it behaves as described. You can use the testing to answer the following questions:
- Does the add-on act as described by the summary and description?
If it says it is an ad-blocker, does it do that or does it instead change your search engine and take over your new tab page?
- Is the add-on generally functional?
If you click on the browser action, page action, sidebar, or options page, do the pages load correctly? We don’t reject for bugs or incomplete features, but if the add-on just does not work at all, we’d like to protect from a bad user experience.
- Does logging in using developer-provided credentials give you access to the described functionality?
Developers are required to provide testing credentials if they use a service. Use these credentials to log in, and ensure that the advertised core functionality exists.
- If opt-in/opt-out UI is for data collection presented, does it convey the consequences correctly? Does declining actually inhibit the data collection?
While you could see this during code review, it is sometimes easier to test. Try to trigger the opt-in/opt-out mechanism, and check if the wording informs the user about the consequences.
An opt-in needs to have the default action be to decline, and at the same time it needs to remain possible to use the core functionality of the add-on after the user declines.
Also check the Network panel in the Developer Tools to make sure the data collection is stopped after declining and does not occur before approving.
- Does the data collected by the add-on comply with our policies?
The policies have fairly specific provisions on how the developer must handle data collection. With help of the network monitor, you can verify what kind of data is being sent to third party sites. You should load at least one (very simple) web site and identify additional requests by the add-on.
- Are there any findings from the code review you would like to verify?
If the code made the impression that sensitive data is being sent but you are unsure, you could use this step to trigger the code in question and check if the data being sent is sensitive.
If necessary, you can use the browser console, network monitor and other developer tools to help with functional testing.
Completing the Review
Add-ons must be reviewed completely, noting issues in the review comments field as you find them. We want to avoid sending multiple partial reviews to developers and give them an opportunity to address all policy violations at once in their next submission.
Depending on the severity of the issue, add-on versions can either be rejected immediately or delayed rejected with the policy violations found. If any additional non-policy violation details need to be communicated, it can be done using a reviewer reply.
In order to support you in communicating with developer, and making consistent review decisions, we have prepared a set of Review Decision Guidelines that you should take a moment to read right now. It contains important information on when a rejection is warranted and will explain how to complete the review.
Conclusion
After reading this document you should be all set up to begin reviewing. You now know how to set up a review environment, are well versed in the different phases from add-on submission to completed review, and have gathered knowledge on how to make a well-informed decision on the review outcome.
If you have any feedback on the review process or there is anything unclear in this guide, please get in touch with the admin team. Your comments will be valuable to improve our documentation and make sure you have all the tools you need to complete reviews effectively.
