Mozilla Test Writing Workshop

Background

The Mozilla project has done a decent job of popularizing extension development, both in terms of teaching people how to write extensions and in terms of publicizing it as a method to enter the Mozilla Community. We aim to organize a test writing workshop to do the same thing for automated regression testing on the Mozilla Platform. We want to capture the most important take-aways from these sessions and create a video resource for online use later by students and community members.

Goals

• Elevate regression/automated test composition to a first order activity in the Mozilla community, equal in weight and prominence to extension development
• Bring together people from all across the Mozilla Community to discuss issues surrounding test development.
• Generate a high-quality, engaging product for future use by the community.

Proposed Topics

Some of the things that we're hoping to cover.

• What are the frameworks?
• How do the frameworks work?
• What problems are each testing framework created to solve?
• How do I write tests for each type of framework?
• What are the frameworks not good for? Where do we need to improve them? What new types of frameworks are needed?
• What is the mindset of the "perfect tester", how do you attain it?
• What is the philosophy/strategy of test development at Mozilla? How do you know when a feature or API is fully covered by a test? How do you plan your work? What's the most important type of test to start with?

Brainstorming The Frameworks/Test strategies we might cover

• Reducing a web page to a test case
• "normal" mochitest, browser-chrome mochitest, chrome mochitest
• XPCShell
• Javascript testing
• Memory Leak testing
• Performance Testing
• Reftests

Format

Our primary goal with this event is to create a reusable artifact that can be integrated into our existing MDC content and serve as a starting point for individuals that are interested in testing. It will also be a perfect starting point for developers that want to get involved with the Mozilla Project but are not ready to leap into feature/patch development yet. So, we really want to focus on bringing together the best minds of Mozilla test development for discussions and how-to workshops.

We do encourage and want an audience, but we are targeting these sessions and the material at people who are already thinking about Mozilla and want to get involved. We are not going to use our time talking about the project in general, instead we will focus on the targeted area of test development.

Also our primary motivation for this is to generate an engaging and informative (and as entertaining as possible) product from the interactions of the presenters and participants. So, it isn't an event for spectators, everyone will be participating at some level.

Logistics

• Dave Humphrey has offered us the use of the Seneca University facilities
• He's also volunteered the University film department to run the videography for the event
• Should it be a one or two day event? We are leaning toward a two day event - one of presentation and discussion, one of hands-on/how-to material.
• Should it be on a weekday/weekend day (i.e. friday/saturday) or on two weekdays?
• We don't want to generate large hours of material that would not be useful to people, so we want to have a very good idea of what and how to film the sessions. We don't want the "videotaped lecture" style of doing this.
• With the impending release of Firefox 3 and the constraints of the school year, it makes the most sense to schedule this event in mid-late September or early-mid October of this year.

Action Items

• Figure out the weekday/weekday vs weekday/weekend day question --> we are going to see if we can blend this into mfinkle's dev day perhaps condense the test material to one day.
• Figure out the number of people that would be interested in attending as participants (i.e. audience). Feel free to add yourself to the sign up sheet below.
• Finalize and invite the core presenters we want to come. Some early thoughts include members of the layout development team, folks responsible for maintaining and expanding the test harnesses, members of the Mozilla QA team, and other community members, and members from other projects. We really want to get a good cross-section of the people who are already involved in it. This event will depend on having the right people present and participate.
• Pitch to other Mozilla projects: Songbird, Miro, Thunderbird, A11y, Calendar, Seamonkey, Camino to see if they have members that want to attend.

Possible two day schedule (with some proposed topics)

We have six possible blocks of time to use on our breakdown of the proposed schedule. We have also discussed dividing the tests into front-end and back-end categories as each requires a slightly different mindset and a different suite of tools for testing.

Some thoughts on the rough schedule above

In order to squeeze this down to one day - I think we want to keep our 2 hour block sessions intact, so we will have to cut content. I think that we can do this by ensuring that each session would address each of the specific concerns within its "world view". For example, the outline of the Front End workshop could go something like this:

• Front End Workshop
  • General idea of mochitests, some front-end test ideas
  • Demonstrate putting together a simple mochitest
  • Talk about case studies of how bug X, Y, and Z were addressed in mochitest
  • Talk about methodologies for other specialized front end tests like reducing a site to a reftest.
  • Talk about what to test a front end for when looking for security bugs, crashers etc.

By breaking up each section into concrete subsections, it gets a little easier to see what we would be filming, and it will help us to capture small, reusable bits of demonstrations which work better than long videos of entire presentations. Also, short subsections like this will help to encourage more participation and interaction among the audience.

So, a possible one-day schedule could look something like this:

• TODO: Which to do first: Front End Testing or Back End Testing?. On one level, starting with Front End testing and moving to Back End testing seems like the natural progression (from the familiar to the unfamiliar). However, I think that starting with Back End testing and moving to Front End testing is an easier progression to learn because the tests themselves and the technologies build on each other better when learned in that order (INMO).

Sign Up Here If You Are Interested

Where will these videos live?

• Here's an idea - put them as part of the Mozilla University effort
• We can create an air mozilla channel and possibly house them there

Current Status (as of May 29, 2008)

We had another meeting to firm up the dates for the conference and to talk more about how the Test Writing Workshop will be integrated into the "Taking Mozilla To the People" event which is being organized by Mark Finkle. The TMTP event is aimed at tapping into IT experts and software engineers to get people thinking about using Mozilla as a platform for both web and thick client development. This event (and it's title) is still a work in progress.

This event will [probably] have two tracks and a breakout room, so it will need three spaces. It will [probably] also make use of the film crew to generate reusable content that we can post onto the web after the conference (just like we are doing for the Test Writing Workshop).

We are thinking that we will be doing this combined event during the week of September 15th, and Dave Humphrey is checking on space and logistics at Seneca University (Toronto, Canada) for that week.

Mark is fleshing out the ideas and the structure of the TMTP event.

The two events are now going to be joined into a two day event. That means that the rough schedule above for a one day Test Writing Workshop is what we will be using as our framework for the schedule.