EngineeringProductivity/Meetings/2010-12-27

Format for today

Let's have a discussion around Goals for 2011 Q1 first. Then, if we have time we'll go through quick statuses (stati?).

Proposed Goals

Let's have a look at where things are.

Outside factors

• Ffx 4 is aligning toward a Q1 ship
• Ffx.next is going to move to Quarterly releases
  • What tools will be needed to manage that?
• E10S is waiting in the wings, maybe ffx.next, maybe ffx.next.next...they desperately need responsiveness measurements in order to make that decision
• Android is marching asymptotically closer to "done-ness" for Talos. Unit tests next...
• Need to start work on Bugzilla (whatever that means). Lots of expectations here, we need to put some stakes in the ground and declare what we'll do first, second, what we will consider in the future and what we won't etc.

Ongoing (Less-negotiable) Projects

• Android infrastructure for talos, unit testing
• War on orange
• E10S infrastructure support
• bughunter
• Pulse?
• Mozmill 2.0
• Addon Testing Service
• Bugzilla/project web dashboards

New (Possible, Highly-Negotiable) Projects

• Replacing ts_cold
• Responsiveness measurements (E10S)
• Replacing/expanding capabilities of "starring" and tree mgmt on TBPL in order to eliminate Tinderbox
• Create our own test system for our own code (devicemanager, profile manager etc etc)
• Bugzilla Extensions (pulse, splinter, etc)
• Automating JS Shell tests on Android
• Investigating how we can aid cjones's new performance testing system development

Meeting Notes

These are the notes for the meeting (we used the above as an agenda).

Q4 Projects Coming in "under the wire"

• Bugzilla/Project/blocker web dashboards - just about complete. Waiting for more feedback before officially moving them into the done category.
• Pulse - we've aided Christian about all we could. He released a production version of pulse this quarter. The next way we can help the pulse effort is to drive the pulse bugzilla extension effort to completion. Work on pulse itself is over for now.
• Android Talos -- looking pretty good. Almost up for an entire weekend unattended except that the Android nightly failed to build and caused talos boxes to burn.
• Mozmill 1.5.2/buildbot - still working on issues surrounding makefiles and 1.5.2 features, but should be resolved in early Q1.

Ongoing Projects

• War on Orange
  • Need to finish up on UI polish, good progress has been made about 50 bugs left (out of 100), should be complete in first few weeks of Q1
  • Need to move atop the metrics provided Db. Initial attempts look good, but we don't know what we don't know. Estimate 1 month
  • Using war on orange Elastic Search Db to aid with eliminating tinderbox (joint goal with releng, gives us a way to contribute to their goal and get more involved with the TBPL tool).
  • We should also mark some time to drive a "crit-smash" style meeting with developers to actually execute on the war on orange effort so that the system we built is used. Probably will be about 5hrs/week for the entire quarter in helping track down issues, answering questions, running meetings etc.

• Bughunter
  • Continued support of existing system
  • The production system has grown too large for couch to handle. Need to move Dbs.
  • Estimate 1-2 months

• Mobile Infrastructure
  • Android
    • Talos should go live in early Q1, next for inclusion in buildbot system will be unit tests
    • Expect to see more stability/reliability issues and ongoing support of releng as they "productize" their buildbot infrastructure to work with the tegras. Given existing track record and rate of product/platform changes, we estimate that we'll be supporting releng in this effort well into Q2 2011.
    • Need to remote the js trace tests for running on android, estimate 1mo work, unknown amount of time for buildbot inclusion
  • Maemo
    • Once we see stabililty/reliability achieved with tegras, will work to move maemo mobile infrastructure to the remoted infrastructure, probably post Q2/late Q1, depending on releng progress with stability on android tegras

• NTT (Nightly Tester Tools)
  • Released new version last week
  • Continued support due to product changes, but no new features planned for Q1 at this time.

• Mozmill 2.0
  • Want to release 2.0 in some usable form this quarter.
  • Team needs to take this offline and figure out what can be done, what the feature set will be etc --> ctalbert will take point to get team together for this.

• Addons Testing Service
  • We realized there will be continued work here but we were uncertain what plans were as Alice was out.

New Projects

• Manifests for test harnesses
  • Mozmill is done, xpcshell is at 75%.
  • What do we want to tackle next? Probably a mochitest flavor, but there's also jetpack, crossweave, etc. Doing mochitest could have the most impact but it is also the hardest one to do. Might start with mochitest browser-chrome.
  • Jmaher to start discussion on what next harness to "manifest" and then we'll be able to estimate effort required

• Bugzilla
  • Dave hit the ground running last week meeting with stakeholders, learning the way around.
  • Thinking about a couple of high level Q1 things: splinter and pulse bugzilla extensions, upgrading b.m.o to Bugzilla 4.0.
  • Also high on the list are a bugzilla-native REST api, branch data support, and true component watching

• CrossWeave
  • We have been asked to provide cross weave on mobile.
  • No current estimates on what that entails yet: jgriffin will suppy those.
  • Is this for maemo or android or both? Is one more important than the other?

• Crowd-source testing for Fennec (Grafx bot style testing with mochitest-browser-chrome)
  • This was mentioned, we are still wondering if we want to do it, and if so what the priority of this would be.

• E10S
  • Ongoing work with special powers extension (bug fixing etc)
  • Determination of how to do user-responsiveness measurements, how to integrate that with talos
  • Porting of tests to "special powers" architecture (Estimate: maybe 2 weeks with developer help)
  • May also need to port "chrome based" tests to work with E10S, issues were found with chrome based tests that load content pages when doing the E10S work for Fennec, we will probably hit that again, and so porting tests may be > 2weeks of effort.
  • May want to refactor mochitest IPC api's used by the mochitest harness to use special powers as well
  • Dependencies: E10S builds of Firefox (for chrome level tests), developer aid in test porting, better design of what we should do for user-responsiveness measurements.

• Performance work
  • Replacing talos ts_cold with something that actually measures cold startup properly on all three platforms
  • Aiding cjones with new performance automation system? This might be too abstract for us to start on in Q1, maybe Q2?

Things missing

• Ted was out, so there are likely some build system goals that he has that we should factor into this discussion as well.
• Likewise, Alice was out and she always has a few more Talos improvements that we should factor in as well as improvements to the Ateam staging system to help us be more effective.
• We'd like to find a way to make our own "Ateam tools" test system so that the tests for our own tools are run, so we may need to spend some cycles on that this Quarter to make that a reality.