QA/Thunderbird3/TestResults/Alpha1/Discussion

2008-05-02, 10am - 11am PDT, #qa

Attendees: davida, dmose, nth10sd, wsmwk, marcia; and Tomcat, tracy and coop later.

Follow-up bugs filed: bug 431883, bug 431884 and bug 431885.

[09:58am] davida: nth10sd: are we doing a phone call as well, or just IRC?
[09:58am] nth10sd: IRC, i was thinking
[09:59am] davida: wfm.
[09:59am] davida: So my big question about the test plan that you have is things like:
[09:59am] nth10sd: so we have me davida and marcia
[09:59am] davida: 1) how long does each kind of test take? (smoke, BFT, spot check)?
[09:59am] davida: 2) how many people are on hand to do it, or is just you?
[10:00am] davida: 3) where does QA for things like web copy get described & done?
[10:00am] davida: but before we get to those, a bit of level-setting.
[10:00am] nth10sd: i'll go first
[10:01am] davida: I think this is an alpha, and in my opinion the expectation of quality is quite low. 
[10:01am] nth10sd: 1) smoketests are loading and starting up the app in each platform, and some basic click-arounds
[10:01am] nth10sd: 2) BFTs are the ones in litmus and because we don't have any tests for TB trunk, I'll be adapting some for trunk from TB 2, and selectively do them since I can't possibly complete everything on my own
[10:02am] nth10sd: spot checks i believe are ad-hoc tests i believe, but marcia can correct me
[10:02am] marcia: nth10sd: yes, spot checks are just running through a quick operations to make sure main things work
[10:02am] nth10sd: the real 2) I am on hand to do all those, because there's nobody in MoMo's QA dept
[10:02am] davida: nth10sd: note that I was asking a specific q about elapsed time.
[10:03am] davida: nth10sd: I don't think QA needs to be done only by people working for MoMo.
[10:03am] nth10sd: marcia: you have an idea on the time taken
[10:03am] davida: In fact I will assert that QA can't just be done by MoMo staff/contractors.
[10:03am] marcia: nth10sd: it depends on who is doing it. Some folks are faster if they are familiar with all the operations
[10:03am] nth10sd: davida: we can't force volunteers to do this tests
[10:03am] davida: nth10sd: no one said anything about forcing.
[10:03am] nth10sd: they can do it if they like
[10:03am] nth10sd: but we must ensure that they are done
[10:04am] davida: for a final or mass-market release, i agree.
[10:04am] davida: for an early alpha, i'm not so sure.
[10:04am] nth10sd: but it
[10:04am] wsmwk_away is now known as wsmwk.
[10:04am] marcia: davida: the alpha will probably get some press coverage
[10:04am] nth10sd: but it's bad PR if for e.g. something screws up for a platform we didn't test
[10:05am] marcia: since there hasn't bee a tbird release in a while
[10:05am] nth10sd: (and this is the first alpha by MoMo)
[10:05am] davida: marcia: i know, but I'm not convinced that one person working 1 day or 5 days will make any difference to the PR impact.
[10:05am] nth10sd: this can easily throw up obvious blockers
[10:05am] davida: I suspect that the way to deal with the PR risk is to message it carefully.  As in "this release will explode".
[10:05am] nth10sd: (through smoketests)
[10:05am] marcia: I agree with gary that I would not want to ship an alpha with major functionality broken on a platform
[10:06am] marcia: we have never done that in the past and we want to keep our streak going
[10:06am] davida: marcia: i agree w/ that "major functionality" point.  I don't mean to say we shouldn't do _any_ testing.
[10:07am] davida: Which is why i was asking about time spent for a BFT for example.  If it's a couple of hours per platform, that seems well worth it.
[10:07am] nth10sd: i'm of the opinion that we should be comprehensive on all our platforms
[10:07am] davida: if it's a week per platform, it's not.
[10:07am] marcia: davida: the BFT would likely take longer than just a few hours. It involves a lot of setting up accounts, etc
[10:07am] marcia: sometimes when i ran the Tbird FFT it could take 2 days
[10:07am] nth10sd: I believe XP has the largest number of users and that's why we test on it
[10:07am] davida: but FFTs are longer, right?
[10:07am] marcia: davida: Yes, FFT are quite a bit longer
[10:08am] wsmwk: FFT?
[10:08am] nth10sd: Full Functionality Tests
[10:08am] Tomcat: full functional test
[10:08am] marcia: nth10sd: I agree that Vista and XP need to work correctly since they will be most visible
[10:08am] nth10sd: (on Litmus)
[10:08am] davida: And the reason I'm being pushy about this is that the current stated tree closure policy is "until we release".  If testing takes a week, then I'm going to push w/ dmose that we branch, so we can unfreeze the tree.
[10:08am] nth10sd: (dmose is coming)
[10:09am] nth10sd: and I can only do so much during the couple of days post-build and pre-release
[10:09am] wsmwk: can we ascertain anything about stability from crash-stats, eg MTBF?
[10:09am] nth10sd: MTBF? (my turn...  )
[10:09am] davida: mean time between failures
[10:09am] wsmwk: mean time between failure
[10:10am] nth10sd: ah ok
[10:10am] nth10sd: wsmwk: look at the graph shape?
[10:10am] davida: wsmwk: good question.  Especially because from what I read somewhere we have a few thousand people using nightlies.
[10:10am] wsmwk: that's probably the only quantitative measure available to us.
[10:11am] nth10sd: except the rate of trunk bugs filed in Bugzilla as well
[10:11am] dmose joined the chat room.
[10:11am] nth10sd: dmose: so we were on the topic of the testplan: http://wiki.mozilla.org/QA/Thunderbird3/TestResults/Alpha1
[10:12am] dmose: ok
[10:12am] davida: marcia, given that you know probably the most about how long it takes to run these things -- how long do you think it would take nth10sd to go through that list?
[10:13am] marcia: davida: Let me look at the list
[10:13am] nth10sd: dmose: and about smoketests on all platforms, with BFTs on XP, and spot checks on the rest
[10:13am] nth10sd: marcia: i adapted it from TB 2
[10:13am] nth10sd: (Fx 3 test plans didn't really seem to be relevant)
[10:13am] davida: My inclination right now is that if nth10sd wants to run a fairly long test plan (say a week), I actually don't mind, as long as we unblock the rest of the dev team, which means branching, which means more work for dmose in particular.
[10:14am] dmose: it's not a ton of work
[10:14am] wsmwk: unsure why we would expect a long test.
[10:14am] marcia: davida: I think the smoketests could probably be completed in 1/2 day. If he going to run BFTs then I would give a buffer of another day or two
[10:14am] dmose: though yesterday we talked about a theory of keeping this short and sweet
[10:14am] dmose: so i'm confused about what's changed
[10:15am] wsmwk: is anytjing more needed than 2-5 ppl running litmus / platform
[10:15am] nth10sd: i haven't done all the tests myself though i'd expect around 2 - 3 days to promise I can complete them
[10:15am] davida: Maybe nothing has changed, but I don't know how to interpret the time impact of gary's test plan.  (hence my question to marcia about timing)
[10:16am] nth10sd: wsmwk: Litmus is good for the community but here we are guaranteeing that they have been done
[10:16am] davida: I've learned that I suck at estimating QA test times 
[10:16am] wsmwk: but litmus is only helpful to the extent that you introduce variability - so > a few is needed.
[10:16am] nth10sd: wsmwk: we know we can rely on you, but we don't know how many of our community will turn up
[10:16am] nth10sd: hence the basic stuff that i must get completed
[10:16am] marcia: Is there a plan to have a test day to bang on the candidate build?
[10:17am] nth10sd: marcia: yes, and someone to operate them as well
[10:17am] marcia: nth10sd: I edited your test plan to cover Tiger and Leopard since they often have different bugs
[10:17am] nth10sd: (if wsmwk doesn't mind, I can let him do it while I focus on the testplan)
[10:17am] nth10sd: marcia: thanks!
[10:17am] wsmwk: I think if you put it out to the community there will be some volunteers
[10:17am] nth10sd: (so that's one platform more)
[10:18am] marcia: at one time we had a mailing list for people interested in testing thunderbird
[10:18am] wsmwk: doesn't need much, but need more than 1-2/per platform.  you could also look to bz QA for a model, not just FF qa
[10:18am] nth10sd: wsmwk: ppl are going to test on the platforms they like but that doesn't mean that we should *not* test a platform our side at all
[10:19am] wsmwk: where is our litmus stuff?
[10:20am] nth10sd: wsmwk: litmus trunk tests don't exist except accessibility ones and one other stuff
[10:20am] nth10sd: i adapt from TB 2, because trunk ones haven't been written
[10:20am] davida: nth10sd: but wouldn't tb2 tests apply 99%?
[10:20am] nth10sd: davida: we have new features like tabbed messaging
[10:20am] davida: in fact figuring out where the tb2 litmus runs fail on trunk would be really useful knowledge.
[10:20am] nth10sd: (that aren't covered in trunk)
[10:20am] wsmwk: mb I'm misunderstanding - bugzilla has a test checklist (forgets if it's litmus)
[10:20am] davida: nth10sd: I wouldn't bother testing tabbed messaging, because IMO it's broken 
[10:21am] marcia: nth10sd: There seems to be a trunk version of Tbird it is just not in the recommended list
[10:21am] nth10sd: as well as things like Gecko 1.9 widgets in Thunderbird
[10:21am] dmose: nth10sd: since this is an alpha (and most especially our first alpha) we have an extremely low bar here
[10:21am] marcia: Need to make sure the crash reporter fires on all platforms since you will want that data in an alpha
[10:21am] dmose: nth10sd: the bar should be "is it basically usable"
[10:22am] nth10sd: so - no loss in major functionality - no major / obvious crashes
[10:22am] dmose: if it crashes in something that's not likely to be used a lot in a day-to-day use, i'm happy to release note and move on
[10:22am] dmose: and tabs is so hard to discover that i probably wouldn't even relnote
[10:23am] wsmwk: yeah, tabs is not ready for QA
[10:23am] nth10sd: the testplan of which is in the wiki that fits in my objective of comprehensive, no loss in major functionality and stable
[10:23am] wsmwk: like dmose's definition
[10:23am] nth10sd: (and i'm not referring to tabs here, point taken about tabs)
[10:24am] dmose: sounds reasonable
[10:24am] nth10sd: so my question now becomes 1) the test plan for our side, 2) the suitability of litmus testcases for the trunk and 3) the running of the testday
[10:24am] dmose: marcia: so right now, we don't have symbols on mac, and we're (at the moment) blocking on it
[10:24am] dmose: marcia: however, i wouldn't want to block on that for too long
[10:24am] marcia: dmose: That is good. Mac is most likely to crash
[10:25am] dmose: marcia:  ie if we don't have any better handle on it by early- to mid- next week, i'd inclined to ship anyway
[10:25am] davida: +1
[10:26am] wsmwk: given that it's alpha, i don't know that we'd block on anything other than what is already flagged + plus what someone mentioned qa functions working like breakpad
[10:26am] dmose: wsmwk: so the symbols thing is that
[10:26am] dmose: wsmwk: right now, no breakpad on mac
[10:26am] dmose: wsmwk: and that's the only currently-plussed blocker
[10:27am] dmose: and i'd rather ship next week than wait even longer to get breakpad on mac
[10:27am] dmose: non-ideal though that may be
[10:27am] nth10sd: https://bugzilla.mozilla.org/show_bug.cgi?id=411171
[10:27am] firebot: nth10sd: Bug 411171 nor, --, ---, rick.tessner@gmail.com, NEW, Thunderbird Mac tinderbox crashing in dump_syms
[10:27am] wsmwk: is that only tinderbox and not crash-stats?
[10:28am] dmose: breakpad == crash-stats == symbols, in this conversation
[10:28am] marcia: nth10sd: https://litmus.mozilla.org/show_test.cgi?id=5118 can be used to test Breakpad functionality for Tbird
[10:28am] dmose: meaning that crash-stats would have info for windows & mac
[10:28am] dmose: er
[10:28am] nth10sd: so with the testplan (and the addition of Tiger) I can safely assure that I can complete the testing by 2 days, if not 3.
[10:28am] dmose: windows & linux
[10:28am] dmose: not mac
[10:29am] nth10sd: (which is pretty weird since Gecko 1.9 Cocoa widgets should have more data)
[10:29am] nth10sd: marcia: true, but that's Firefox, though it can be adapted to TB
[10:29am] marcia_leopard: nth10sd: I just tested it, it installs fine into Tbird and crashes it on mac
[10:29am] nth10sd: (someone should write TB trunk testcases in Litmus if we are going to use Litmus for the future)
[10:30am] nth10sd: marcia_leopard: so it works properly?
[10:30am] Aleksej: Are crash reports processed today?
[10:30am] dmose: no
[10:30am] dmose: crash reports on mac are currently busted
[10:30am] marcia_leopard: nth10sd: yes, it does on mac at least
[10:31am] marcia_leopard: Aleksej: The discussion above is about Thunderbird crash reports, not Firefox in case you are wondering
[10:31am] Aleksej: marcia_leopard: I haven't looked at it 
[10:31am] Aleksej: My todays Firefox crash reports are not processed yet 
[10:31am] marcia: Aleksej: 
[10:32am] nth10sd: so, back to my question:
[10:32am] nth10sd: 1) the test plan for our side, 2) the usage of TB 2 litmus testcases for the trunk and 3) the running of the testday
[10:32am] wsmwk: if symbols busted that long then not a blocker + we can move on  + back to gary
[10:32am] nth10sd: is this a decent plan?
[10:32am] nth10sd: (back to gary.  )
[10:33am] marcia: nth10sd: I think your test plan is fine
[10:33am] marcia: the litmus test cases that are can be used as a framework
[10:33am] nth10sd: so now we set the QA plans as the foundation for the future alphas, at least
[10:34am] wsmwk: if we want to improve litmus for future, gary what would you want to see hqppen?
[10:34am] nth10sd: wsmwk: someone must write Thunderbird trunk Litmus testcases
[10:34am] nth10sd: make that Thunderbird Trunk-specific testcases that guides new testers along
[10:35am] wsmwk: + identify cases needed?
[10:35am] dmose: nth10sd: this sounds like something that could be done on test-writing days
[10:35am] nth10sd: dmose: idea for the future
[10:35am] nth10sd: wsmwk: hang on
[10:36am] dmose: nth10sd: additionally, as we move towards a more test-driven development model, we should start encouraging devs to write tests (ideally in a suite, but litmus is a good fallback) for bugs as they fix them
[10:36am] dmose: and not just devs, really, anyone who's interested in doing that work
[10:36am] dmose: getting triagers involved at the level, for those have the skillset, would be great
[10:36am] davida: yeah, it would be interesting to see what % of litmus tests could be converted to automated tests.
[10:36am] nth10sd: wsmwk: if you log in to litmus and click view/search tests, you select Thunderbird then Trunk
[10:37am] wsmwk: i'm just thinking that, if litmus is to be a key QA item for future, if we get trunk users to buy into using it, we can use there input as to what needs improvement, as well as another way to get them involved.
[10:37am] nth10sd: you'd see that Testgroups only consist of accessibility and l10n
[10:37am] wsmwk: but as david says we don't need it as a basis for releasing a1
[10:37am] nth10sd: I am of the opinion that Litmus is a key QA item for the future, but I won't count on getting users to test a1 now
[10:38am] dmose: agreed; litmus is a great way to test stuff that's not yet automatically testable
[10:38am] nth10sd: s/to test a1 now/to test a1 using Litmus now, if they are new testers
[10:38am] nth10sd: but we are laying the groundwork for the future alphas
[10:39am] davida: i'm not sure we have resolution on the timing issue that I'm most concerned about.
[10:39am] davida: speaking egotistically 
[10:39am] dmose: the timing issue being what, exactly?
[10:39am] nth10sd: davida: i mentioned with the testplan (with Tiger thrown in) I can get it completed by 2 days, if not 3
[10:39am] davida: well starting with t=0 being when a build is available, what is a reasonable estimate for QA signoff in the optimistic case that no blockers are found?
[10:40am] nth10sd: (I can guarantee safely saying that I can get it completed myself, is 2-3 days)
[10:40am] davida: nth10sd: but isn't there a test day also in the loop, and can that be done in parallel w/ little advance notice?
[10:40am] nth10sd: but with that, i
[10:40am] nth10sd: i'm not sure if i have much bandwidth left for testday
[10:40am] davida: I also don't like the idea that you, nth10sd, feel that you're personally responsible for holding up a release.  that seems wrong.
[10:40am] nth10sd: i'm sure wsmwk can help with testdays
[10:41am] nth10sd: i'm just trying to ensure that everything is as comprehensive as possible
[10:41am] dmose: yeah, but pinning that all on you doesn't scale
[10:41am] nth10sd: "holding up a release" seems an inappropriate phrase
[10:41am] wsmwk: testday OK. but more to the point is, what would a testday reveal that we would block on? if there is nothing, then do we need a testday?
[10:41am] wsmwk: prior to release
[10:41am] dmose: well, if the testday's goal was running through the testplan, i'd hope so!
[10:41am] nth10sd: testdays are for ensuring that any potential blockers are discovered pre-release
[10:42am] dmose: or, at least, making as much progress in the test plan as possible
[10:42am] nth10sd: wsmwk: i could assign a you a platform on whatever you're comfortable with
[10:42am] wsmwk: if to validate a testplan then that's a good thing.
[10:43am] wsmwk: but if we are on timed release, then the goal is not primarily to find blockers.
[10:43am] dmose: i guess i might be using "testday" in a way that's slightly different than the traditional ussage
[10:43am] nth10sd: generally, historically, basic smoketests are done by Mozilla (whatever company)
[10:43am] dmose: meaning "some official day that we attempt to engage the testing community to help with whatever we need done for release"
[10:43am] davida: nth10sd: forget about that history 
[10:43am] nth10sd: hmmmm
[10:44am] dmose: yeah, the employer of record of the folks who do the smoketesting is not interesting
[10:44am] davida: nth10sd: i mean it.  i don't have a budget for a full QA department.  so we have to find ways to use volunteers, and adjust everything as approprite.
[10:44am] davida: even if it means changing our critiria for release.
[10:44am] dmose: the interesting bit is that we feel confident that whoever is doing it is doing a reasonable (but not perfect!) job
[10:44am] • wsmwk wonders where are the Mac-fans are that symbols are not working for such a long time?
[10:44am] nth10sd: so we move away from the way we used to do it, the way that has ensured our QA for the past releases / reputation?
[10:44am] davida: criteria
[10:44am] nth10sd: hmmmm ok
[10:44am] dmose: wsmwk: it's only been two weeks
[10:45am] dmose: nth10sd: this is an alpha, our reputation is not staked on alpha releases
[10:45am] wsmwk: but missing symbols seems to happen with some unplanned frequency
[10:45am] davida: nth10sd: I personally believe that the reputation for quality is the result of much more than things like running through litmus, but that's another conversation.
[10:45am] dmose: wsmwk: yeah, it seems to be a bug in dump_syms; hopefully this will make that stup
[10:45am] nth10sd: davida: yes those are 2 things
[10:45am] davida: wsmwk: it looks like a heisenbug
[10:45am] dmose: davida: agreed; just having nightly test builds is where tons of our quality comes from
[10:46am] wsmwk: agree
[10:46am] wsmwk: we are building quality, not guaranteeing
[10:46am] nth10sd: so we still have yet to agree on the testplan for the alpha
[10:46am] nth10sd: 1) the test plan for our side, 2) the usage of TB 2 litmus testcases for the trunk and 3) the running of the testday
[10:47am] wsmwk: i'd say have a testplan, go through the motions of a testday, but have an extremely high bar not t release
[10:47am] nth10sd: and get volunteers to run TB 2 litmus tests on trunk?
[10:47am] wsmwk: if that's part of the testplan, yes
[10:47am] dmose: that sounds right to me
[10:48am] davida: but if no one shows up, oh well.
[10:48am] dmose: marcia: so does the wiki page seem like it's a reasonable test plan?
[10:48am] davida: in the short term oh well.  in the long term we need to fix that.
[10:48am] • nth10sd notes that marcia has approved as well
[10:48am] marcia: dmose: Yes, I confirmed that earlier
[10:48am] nth10sd: (backscroll?)
[10:48am] nth10sd: yup
[10:48am] tracy: y'all can get coop to do a staright copy of the TB2 test cases into a TB3 suite. then work on cleaning those up as they apply to trunk builds.
[10:49am] davida: tracy: good idea.
[10:49am] dmose: marcia: ok, great
[10:49am] dmose: nth10sd: so it looks like we've got an agreed upon test plan, then, no?
[10:49am] tracy: davida: let me know it you need any help facilitating that.
[10:49am] nth10sd: yes
[10:50am] davida: tracy: i have no idea what's involved, so i probably will
[10:50am] nth10sd: the testplan has only been modified to add Tiger at http://wiki.mozilla.org/QA/Thunderbird3/TestResults/Alpha1
[10:50am] nth10sd: and the consensus that we will be using Litmus for volunteers at the a1 testday
[10:50am] dmose: nth10sd: sounds good
[10:50am] davida: also that the a1 test day will be scheduled as soon as the builds are ready?
[10:51am] nth10sd: davida: yes
[10:51am] nth10sd: we are holding on when Rick gets the builds out
[10:51am] dmose: we can just schedule a test day by saying "2 days from now", right?
[10:51am] nth10sd: yes
[10:51am] dmose: i wouldn't want to have to wait until some specific thursday
[10:51am] nth10sd: once he gets the builds out wsmwk and i can easily quickly advertise a testday
[10:52am] wsmwk: yup
[10:52am] nth10sd: since we had been doing this for weeks
[10:52am] dmose: ok, great
[10:52am] nth10sd: dmose: ideally on thursday but we can always change
[10:53am] dmose: nth10sd: so in some basic way, i think we've covered all three of your most recent questions, though not in a lot of depth.  is there anything we need to nail down further?
[10:53am] nth10sd: wsmwk: you think you can guide volunteers to litmus on testday?
[10:53am] wsmwk: yes
[10:53am] nth10sd: (i will still be on for session 2)
[10:53am] nth10sd: ok
[10:53am] nth10sd: great
[10:53am] nth10sd: then i think i'd reiterate:
[10:54am] wsmwk: the only question from me is, how do we gather info about what needs to be added/changed in litmus - is failed litmus test sufficient?
[10:54am] nth10sd: 1) the test plan for our side,(OK) 2) the usage of TB 2 litmus testcases for the trunk (YES) and 3) the running of the testday (USE LITMUS)
[10:54am] tracy: historically, short notice for testdays hasn't worked out so well.  which is why we've stuck to a schedule and really tried to have a list of future testday topics available well ahead of time.
[10:54am] nth10sd: tracy: i wouldn't hold on that
[10:54am] nth10sd: marcia: how do we deal with failed litmus testcases?
[10:54am] dmose: tracy: hmmm, interesting
[10:54am] dmose: we could simply schedule a test day for thursday
[10:55am] nth10sd: dmose: and use a nightly?
[10:55am] marcia: nth10sd: periodically we just review the failed ones and make sure they are really fails. SOmetimes people fail just on verbiage
[10:55am] tracy: dmose, that's what I'd suggest.
[10:55am] dmose: if we don't have a blocker, yeah
[10:55am] wsmwk: agree with gary, if the bigger goal is to test the process, not the product.
[10:55am] marcia: nth10sd: there is automated way to get failed results from Litmus on a daily basis
[10:55am] nth10sd: ah
[10:55am] marcia: coop does it for Firefox, so I am sure it can be adapted to Tbird
[10:55am] dmose: s/blocker/build
[10:56am] tracy: nth10sd: testday reports can also be setup specifically for your testday.
[10:56am] nth10sd: wow so many things still to be done for Litmus
[10:57am] nth10sd: Litmus issues: 1) morph TB2 to TB3 testcases, 2) automated way to get failed results from Litmus on a daily basis and 3) set up testday reports for Litmus
[10:57am] nth10sd: can coop do all those above?
[10:57am] nth10sd: s/do/help us with
[10:58am] coop: if you've setup your testdays, the reports are automatic
[10:58am] coop: and please file bugs for the first two
[10:58am] • nth10sd wonders how to setup testdays
[10:58am] tracy: admins can setup the testday report..
[10:58am] nth10sd: ok, i'll file the bugs
[10:58am] nth10sd: otherwise i don't have anything else for the discussion
[10:58am] tracy: I think the daily report covers all, yes?  coop?
[10:58am] Tomcat: i can setup the testday report
[10:59am] nth10sd: dmose davida marcia wsmwk: anything else?
[10:59am] nth10sd: thanks Tomcat
[10:59am] marcia: nth10sd: not that i can think of
[10:59am] coop: tracy: it might right now
[10:59am] davida: nope, gotta be on a call anyway.
[10:59am] Tomcat: nth10sd: the create testday report bug can you assign to me
[10:59am] davida is now known as davida_phone.
[10:59am] dmose: sounds good to me; thanks everyone for helping us sort through this stuff
[10:59am] Tomcat:
[10:59am] wsmwk: sounds good.
[10:59am] nth10sd: thank you everyone!
[10:59am] wsmwk: quick OT question
[10:59am] coop: but if nth10sd is going to handle the thunderbird results, i could split them up
[10:59am] nth10sd: coop: handle results?
[10:59am] wsmwk: do we have symbols on server for thunderbird, eg
[11:00am] wsmwk: http://symbols.mozilla.org/thunderbird
[11:00am] wsmwk: http://symbols.mozilla.org/seamonkey
[11:00am] wsmwk: http://symbols.mozilla.org/thunderbird
[11:00am] wsmwk: http://symbols.mozilla.org/seamonkey
[11:00am] dmose: i got the impression from ted that we might
[11:00am] dmose: but it'd be good to double-check at some point
[11:00am] nth10sd: wsmwk: sigh~ file a bug?
[11:00am] wsmwk: we need that for some future qa work. 
[11:00am] wsmwk: to QA hangs, loops, etc -
[11:01am] wsmwk: wihtout debug builds
[11:01am] wsmwk: i'll check with ted
[11:01am] nth10sd: wsmwk: i'd say file a bug and probably flag it
[11:02am] dmose: wsmwk: or just try it and see if it works
[11:02am] nth10sd: yeah
[11:02am] wsmwk: i'll farm it out 
[11:03am] nth10sd: coop: re: filing the bugs, which components should they be in? and should i cc you?
[11:04am] davida_phone left the chat room. (Ping timeout)
[11:04am] coop: webtools/litmus, and no need to cc me directly, i'll see them