Calendar:QA Chat:2008-01-10:Log
From MozillaWiki
[18:32] <mschroeder> TADA... it's QA Chat time! [18:33] <mschroeder> Our agenda for today's QA Chat: http://wiki.mozilla.org/Calendar:QA_Chat:2008-01-10 [18:35] <nth10sd> mschroeder: what's eggplant? (and the agenda looks similar to the prev week? (which had only u and Andrea) [18:36] <mschroeder> nth10sd: Eggplant is software for automated UI testing (on Mac). [18:36] <nth10sd> i see [18:36] <mschroeder> Andreas: ping [18:36] <ctalbert> Eggplant is actually cross platform, but it is closed source and requires a license. [18:37] <Andreas> mschroeder: pong [18:37] <ctalbert> thetux: you're welcome. That's probably one of the worst bits of UI in the entire bugzilla product. [18:37] <mschroeder> Andreas: just wanted to check you're here for the QA chat. [18:37] <mschroeder> ;) [18:37] <ctalbert> but unfortunately a very important one [18:38] <Andreas> Yes, i'm back. [18:38] <nth10sd> mschroeder: ulf's leaving calendar? [18:38] <mschroeder> nth10sd: Has already left. [18:38] <nth10sd> mschroeder: i see. that's one less. [18:39] <mschroeder> First topic: Test day today!!! [18:39] <mschroeder> We had _no_ Litmus tests today until now. [18:40] <mschroeder> So, if you have time to run especially the drag&drop-conversion tests, do it. :) [18:42] <Andreas> It would be great someone with a mac could run this testcases. [18:42] <skuribay> I'm in d-d testcase... [18:42] <mschroeder> skuribay: perfect [18:44] <mschroeder> Next topic: Refurbishing Litmus testcases [18:45] <mschroeder> I'll try to contact damian again. Hopefully he'll have some spare time. [18:46] <mschroeder> If anyone is interested to help in cleaning up and reworking our existing Litmus testcases, feel free to raise your hand. :) [18:46] <Andreas> I will try to rework some of the litmustest written by Ulf and me. [18:47] <mschroeder> That's a good start. [18:48] <thetux> i would help, but i'm not sure if i'm the right person for this, since i just have an litmus account for a few minutes... [18:49] <mschroeder> thetux: Try do run some Litmus testcases from Calendar group. If you find a testcase that isn't exercisable anymore (e.g. because a dialog or menu changed), add a comment, so we can find it- [18:50] <thetux> ok :) [18:51] <mschroeder> thetux: That already helps a lot. [18:51] <mschroeder> Next topic: QA sessions: qawanted, qa discussion needed, verifying fixed bugs [18:52] <mschroeder> I think we should concentrate on 0.8 testing and verifying fixed bugs at the moment. [18:52] <ctalbert> And ad-hoc testing too [18:52] <ctalbert> oh wait you said that [18:53] <ctalbert> +1 :-) [18:53] <mschroeder> Andreas does a good job verifying bugs. :) [18:55] <mschroeder> Next topic: AI: "in-testsuite" flag for Calendar component in Bugzilla (mschroeder) [18:56] * mschroeder talked to daniel, and I'll file a bug after the QA chat. [18:57] <mschroeder> Next topic: AI: Enable unit tests on tinderboxen (mschroeder) [18:57] <mschroeder> I had no time to make any progress. [18:58] <Sebo> isn't that pending review? [18:59] <mschroeder> Sebo: no, I updated the Makefile on Trunk, but enabling tests on our production tinderboxen is complicated. [19:00] <Sebo> ah, I thought the Makefile will do the trick... [19:01] <mschroeder> no, I have to talk to robcee, who should know what to do next [19:02] <mschroeder> Next topic: Test Automation Sumit [19:02] <thetux> short question: When the steps to perform in litmus are not right, where to write this down? [19:02] <mschroeder> thetux: mark it unclear and add a comment [19:03] <thetux> k [19:04] <mschroeder> The effort has stalled because ctalbert and I have too much on our plates at the moment. [19:04] <ctalbert> mschroeder: well, it's a little different than that [19:05] <ctalbert> I talked with the folks in the Mozilla Automation team about the idea, and they are all for it. [19:05] <mschroeder> sorry [19:05] <ctalbert> that's ok [19:05] <ctalbert> I need to blog abotu this [19:05] <ctalbert> And what I'm starting to think is that it would be more beneficial to every project, if we did a Mozilla-wide automation summit. [19:06] <ctalbert> And if we did that, we would have more help in putting it on than just those of us in the calendar project [19:06] <ctalbert> We could draw several people in that are interested in thunderbird, firefox, camino, seamonkey etc [19:06] * mschroeder agrees [19:06] <ctalbert> And we could do more with more of the automation tools that mozilla offers if we do it that way too - we could cover the reftests, litmus, QAX, mochikit, XPCShell etc [19:07] <ctalbert> So, I think the original idea has changed slightly - it has gotten bigger. But, it's still on the table. I'm just not exactly sure when we're going to actually *do* it. If we hire someone that we just interviewed, then he and I will probably make it our task to spearhead this effort and get it done. [19:08] <mschroeder> cool [19:08] <ctalbert> I'll post something to the calendar blog about this to update folks. [19:08] <mschroeder> ctalbert: Thanks for the update. :) [19:09] <ctalbert> Sure, np [19:09] <mschroeder> Last topic: Automated UI testing: Eggplant & ... [19:10] <Sebo> Is firefox doing UI testing? [19:11] <mschroeder> I talked to celina63 last week, and she had used some tool from Hewlett-Packard at work. She also gave my a link to some tools, but I haven't look at them. [19:11] <ctalbert> Sebo: yes. [19:11] <mschroeder> Sebo: I think, they use Eggplant for Firefox testing. [19:12] <ctalbert> Firefox uses Eggplant and has an entire team dedicated to UI testing. [19:12] <Sebo> wow [19:12] <ctalbert> You've probably heard a mention of the "QA Execution Team" -- that's who they are [19:13] <Sebo> but we would need a license? [19:13] <ctalbert> Since Eggplant is closed source, it's a bit difficult for the Mozilla license to be used by community projects (because MoCo bought a certain number of seats for that license). [19:14] <ctalbert> There has been a lot of talk about this, and people are trying to find a way to do that, though. Tracy probably knows where that effort is at. [19:14] <ctalbert> Sebo: yes, is the short answer :-) [19:14] <Sebo> hehe [19:14] <Sebo> thanks [19:14] <ctalbert> If we could find some other tool that could work for us, that'd be awesome. [19:15] <ctalbert> Many of them are OS specific, but if we could even get coverage on one OS with a tool, that would be OK since it would free us up to look at other OS's. [19:16] <Sebo> what sort of things would you actually test that we can't test otherwise? [19:18] <ctalbert> GUI automation would test stuff that you could test by hand. [19:18] <ctalbert> In fact, it will only find glaring errors. [19:18] <mschroeder> The Calendar views, menus and dialogs. [19:18] <ctalbert> ususally [19:18] <ctalbert> But, it will be able to run through all that stuff a lot faster than you could by hand. [19:19] <ctalbert> And that's why it makes a good "smoketest" tool. Especially useful for l10N testing. [19:19] <ctalbert> When we have to go through all the builds for all the locales right before release. [19:19] <Sebo> yeah, l10n is a good argument [19:19] <ctalbert> that's probably where it is most beneficial. [19:19] <Sebo> regarding the rest I would say - we don't have so much to cover that its worth it [19:19] <ctalbert> but if the UI is funny-looking and unusable, the GUI automation tool won't capture that. [19:20] <ctalbert> Only a human can do that. [19:20] <ctalbert> Sebo: you are probably right [19:22] <mschroeder> Okay, any question, suggestions etc.? [19:23] <mschroeder> You can set the in-testsuite flag now, if you find a bug that needs a unit test. [19:24] <mschroeder> reed created it for me one minutes after filing the bug. :) [19:24] <ctalbert> he tends to do that. [19:25] <mschroeder> That's the end of the QA chat, thanks for participating.