Calendar:QA Chat:2007-11-15:Log

[18:30] <mschroeder> Hi ulf, andreas
[18:30] <Andreas> Hi mschroeder
[18:30] <ulf_> hi mschroeder!
[18:33] <mschroeder> Welcome to the QA chat! See our agenda at: http://wiki.mozilla.org/Calendar:QA_Chat:2007-11-15
[18:33] <Andreas> The 'automation summit' witten by ctalbert is realy cool stuff!
[18:34] <mschroeder> It absolutely is!
[18:34] <mschroeder> Also see following blog post from yesterday: http://weblogs.mozillazine.org/calendar/2007/11/test_automation_summit_plannin.html
[18:35] <Andreas> Yes
[18:37] <mschroeder> No news on unit test on tinderboxen. So we can start to discuss the Test Automation Summit.
[18:38] <mschroeder> alawrence: are you here?
[18:39] <alawrence> here
[18:39] <ulf_> mschroeder: apologies, haven't answered the post 0.7 ng discussion yet. Probably because 0.7 went quite well all in all
[18:40] <mschroeder> ulf: no problem.
[18:40] <mschroeder> Let's have a look at http://wiki.mozilla.org/User:Ctalbert:Automation_Summit and discuss it.
[18:41] <alawrence> mschroeder: I have terse notes about how to get started already
[18:41] <alawrence> they'd need to be expanded some
[18:42] <alawrence> for instance, they already talk about hold to get your new test into the Makefile
[18:42] <mschroeder> alawrence: you can send them to ctalbert and me, and we'll look at it.
[18:42] <alawrence> will do
[18:43] <mschroeder> Back to the discussion: We have to ensure to agree on the rough plan.
[18:45] <mschroeder> First is the preperation of documents. alawrence and I already have some notes.
[18:45] <ulf_> when will the summit take place?
[18:47] <mschroeder> ulf: maybe late January, but nothing has been set in stone yet.
[18:47] <ulf_> ok, see. Thx!
[18:48] <mschroeder> So, we should have a look at the first section of ctalbert's wiki page.
[18:49] <mschroeder> There is existing documentation we can use. But we also have to create an own tutorial and documentation that is tailored for our automated testing effort.
[18:51] <ulf_> mschroeder: in general I'd expect Andreas and me more in the audience of the event than as mentors, because this is all complete new stuff to us, you know
[18:53] <ulf_> we absolutely willing to learn and perhaps also take over little things, but unit tests w. xpcshell is "Neuland" to us
[18:54] <alawrence> the audience for that tutorial would be people with some general development experience, but nothing specific to mozilla or even javascript?
[18:54] <mschroeder> ulf: That's all right. But I hope you comment on the organizational points. :)
[18:54] <ulf_> sure
[18:56] <mschroeder> alawrence: basically we want to catch everyone's attention who is interested and willing to look at doing Mozilla stuff.
[18:56] <alawrence> ok
[18:57] <mschroeder> so we also need the basic programming stuff like variables, functions, control structures
[18:58] <mschroeder> If you look at ctalbert's list of documents we want, you can see them at the top.
[18:59] <alawrence> got it
[18:59] <mschroeder> One side is JavaScript, the other the Mozilla-specific XPCOM.
[19:00] <mschroeder> After that we need to introduce the test framework XPCShell.
[19:01] <mschroeder> The next step is to teach them the Mozilla workflow and style (reviews, code style etc.)
[19:02] <mschroeder> Should we seperate those documents?
[19:04] <mschroeder> Do we want long documents, so you have to scroll a lot?
[19:05] <ulf_> regarding the test framework: what requirements are you thinking off? Will the framework be able to record test results, do test comparisons etc?
[19:05] <mschroeder> There is a test log for every JavaScript file that runs tests.
[19:06] <mschroeder> What do you mean by test comparisons?
[19:06] <ulf_> how is the log evaluated, manually?
[19:07] <mschroeder> If you run it locally, yes. You have the shell and the log reporting failed tests.
[19:08] <ulf_> w. comparison I mean to compare a test result w. expected values
[19:09] <mschroeder> there are functions: do_test_true, do_test_false, do_test_eq(value, expected value), do_test_neq(...)
[19:09] <ulf_> mschroeder: if I understand well a test is either failed or passed, but the complete evaluation has to be done manually post to the test run, right?
[19:09] <mschroeder> ulf: yes, but you can easily write a script to evaluate.
[19:10] <ulf_> so you wouldn't think of comparison algorithms to be part of the framework
[19:11] <ulf_> means, should be handeled in the test case
[19:12] <mschroeder> one testcase has multiple comparison tests. if one of these tests fails, this is written in the log including the values that weren't equivalent.
[19:13] <mschroeder> maybe I misunderstand you.
[19:13] <ulf_> ok, thx for the explanation
[19:14] <mschroeder> Everyone: What is your opinion on question #4 at http://wiki.mozilla.org/User:Ctalbert:Automation_Summit#Open_Questions
[19:14] <mschroeder> ?
[19:16] <ulf_> listener interface, various ideas?
[19:17] <mschroeder> I'm talking about: For the sessions that are not calendar specific, should we get speakers from outside the project to either moderate or present? The best example of this would be the automation/unit test/regression test/ philosophy talk?
[19:19] <mschroeder> Does anybody known an expert in this field we could invite as a speaker?
[19:20] <ulf_> mschroeder: sorry wrong paragraph. External speakers sounds very good. Perhaps from other mozilla projects - no, I don't have names
[19:20] <ulf_> perhaps Clint could engage some colleagues
[19:21] <mschroeder> I have to look for some literature about testing.
[19:21] <mschroeder> Maybe there are some authors we can ask.
[19:23] <alawrence> mschroeder: the only name I can think of currently is Kent Beck  http://en.wikipedia.org/wiki/Kent_Beck
[19:24] <alawrence> I bet he'd give a good talk, but I believe it would be flavored towards Test Driven Development, which is not exactly what were talking about here
[19:25] <mschroeder> But he developed JUnit together with Erich Gamma.
[19:25] <alawrence> right
[19:26] <mschroeder> So, developers can and should use existing testcases to test every time they change anything in the source code.
[19:26] <alawrence> yes
[19:26] <mschroeder> So, unit tests and test driven developement are connected.
[19:26] <alawrence> very much so
[19:27] <alawrence> the difference is this:  Beck, and MANY others advocate writing the tests FIRST, then coding
[19:27] <alawrence> not the other way around
[19:28] <alawrence> this, IMHO, is a great idea, and should be considered for future dev maybe
[19:28] <mschroeder> that's true. but programmers often start coding and don't think about tests until they have a bug they can't find. ;)
[19:28] <alawrence> also true
[19:28] <mschroeder> better: they can't find the cause.
[19:30] <alawrence> and then they can prove they fixed it because the test stays in place, and helps keep it from breaking in the future
[19:30] <mschroeder> Next week we continue our discussion. Maye we have some more attendees then. We have to talk about publicity and how to structure the summit. ctalbert has already outlined his ideas on the wiki page.
[19:30] <mschroeder> alawrence: yeah, that's the idea.
[19:32] <mschroeder> ... Publicity ... If you know people interested in coding Mozilla-based or to help with our whole effort, invite them for next weeks QA chat.
[19:32] <mschroeder> :)
[19:34] <mschroeder> We're running out of time. Thank you for attending. The QA chat is officially over.
[19:34] <ulf_> thx mschroeder; bye!
[19:34] <Andreas> Ok, bye.