Firefox/Input/Thank you page

Thank you page overhaul

Summary

When a user leaves feedback using one of the forms on the site, they are redirected to a "Thank you!" page. Currently, all users see the same page regardless of their sentiment or the contents of their feedback. This page isn't very helpful to anyone.

We see many different kinds of feedback, but there are three kinds of feedback that are interesting in regards to content on the "Thank you!" page:

1. feedback from a user who is very happy with the product
2. feedback that's about a problem that we have known solutions for
3. feedback that's about a problem we need more information on

We want to do things for these groups of people. This project covers that work broken into several phases and experiments.

Status

• Project owner: Will Kahn-Greene
• Status: Phase 1 complete

History

• Since the dawn of time: We've been talking about this and thinking about the kinds of things we could do and what would be most effective for users while simultaneously lamenting a complete lack of time to work on it.
• May 8th, 2014: Will decided to put a bug on the books about this (https://bugzilla.mozilla.org/show_bug.cgi?id=1007840)
• December 17th, 2014: Will wrote up this initial project plan.
• March 9th, 2015: End of OPW. We've got some pieces done, but there's still a bunch left to do.
• June 3rd, 2015: Landed the suggest framework and redirector.
• June 10th, 2015: Pushed phase 1 to prod and made it live.

Phase 1: Completed

Summary

This phase will focus on Input feedback with the following criteria:

1. en-US only
2. Firefox browser product only
3. sad feedback only
4. require at least N words? (theory: fewer than N words will yield lousy search results from SUMO plus they're less likely asking about a problem.)

When this happens, the user will see a new version of the "Thank you!" page which will have 3 links of suggested SUMO kb articles and one "I'd like to ask a question" link which goes to SUMO AAQ.

Measuring success

Bug #1133774 covered figuring out what we want to measure and from that what we need to capture.

Question: Are we helping users?

This is essentially a question of engagement--are users clicking on the suggested links we've provided? To determine whether this is helping users, we need to find out whether users are clicking on the suggested links.

Then engagement is something like:

   clicked on at least one link
   ----------------------------
   total people offered a suggestion

Gregg was thinking that 20% engagement might be a good number to shoot for. If we end up with like 5% engagement, then maybe we're trying to solve a problem that doesn't exist or the heuristics generating the suggestions are sub-par.

Question: Does tailoring work? How attractive are the results?

We can figure this out by comparing:

   clicked on a suggested link
   ---------------------------
   total people offered a suggestion

vs.

   clicked on "None of these helped" link
   --------------------------------------
   total people offered a suggestion

We'll push events to Google Analytics and use the Event Flows view to measure these.

Requirements

1. implement SUMO suggest provider which:
   1. looks at a feedback response and determines whether to generate suggested links or not
   2. generates a request against the SUMO Search Suggest API to retrieve relevant links
   3. returns relevant links wrapped in a redirector url
2. implement SUMO suggest redirector which:
   1. handles SUMO suggest redirect links
   2. posts a GA event
   3. redirect the user to the outbound link
3. new thank you page view and template to show suggested links
4. tests for everything involved

Bugs

Tracker bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1007840

Full Query

ID	Summary	Priority	Status
1083405	Create API for suggested articles and question in AAQ flow.	P2	RESOLVED
1116523	add new response id to session	--	RESOLVED
1116838	add waffle flag for thank you page	--	RESOLVED
1116848	change thank you page view to allow for thank you page variations	--	RESOLVED
1124412	[research] evaluate SUMO search APIs for best results given a piece of feedback	--	RESOLVED
1129615	mockups for thank you page for sad feedback	P2	RESOLVED
1133769	implement sumo search code	P2	RESOLVED
1133771	implement search results display code in thank you page template	P2	RESOLVED
1133774	[research] figure out how to track performance of new thank you page	P2	RESOLVED
1167241	add version to Link in suggest framework	P1	RESOLVED
1169261	implement suggestion link redirector	P2	RESOLVED
1173430	analyze success of Thank You page project phase 1	P2	RESOLVED
1173763	event label for some sumo_suggest1 events is "https:" in GA	--	RESOLVED

13 Total; 0 Open (0%); 13 Resolved (100%); 0 Verified (0%);

Phase 2

Summary

This phase will focus on expanding the existing feedback to cover non-en-US locales and non-Firefox Desktop products.

Measuring success

Bug 1133774 added GA events for measuring the success of the original flow. We can extend these events with custom variables showing the locale and product active when a suggestion is made or chosen, allowing us to see both general and locale/project-specific measurements.

Requirements

1. Update SUMO suggest provider to:
   • Limit search results to a specific language based on Input's current language.
   • Enable search results for other products besides Firefox Desktop, configurable on a per-product basis.
     • Figure out how to maintain a mapping of Input products to SUMO products.
2. Update the SUMO suggest redirector to:
   • Append the locale and product active when the suggestions were generated as custom variables to the GA event.

Bugs

Tracker bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1233792

Full Query

ID	Summary	Priority	Status
1173431	extend sumo suggest provider to cover all locales supported by sumo	P2	RESOLVED
1214275	extend sumo suggest provider to cover all products supported by sumo	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Future phases

Future phases will be figured out after the most recent phase is completed so we have time to act on what we've learned.

Future phases may include:

1. pulling solved questions (plus some other filters?) from SUMO?
2. pulling a mix of solved questions (plus some other filters?) and KB articles from SUMO?
3. get a kb article keyword -> article map for top 25 articles per locale and using that instead of doing lots of searches?