Raw Log from this session:

OK, Our homework was to write down the set of things we thought of to make automatoin and developer's lives better. So, what did you think of.

Problems

• Replication envirionment differences from local to remote
  • Lots of environments to get to and get onto
  • Or downloadable environment
  • At red-hat they had separate way of running tests where upon a failure it would not return the machine to the pool, it would leave it as-is and ping who reserved it/initiated the test run and see exactly where it was at when it failed.
  • But how would that work for the community members? Could they use downloadable solution instead?
  • If we have them virtualized, then we could build a copy w/o keys and give people access to that.
• EC2 - do you get guaranteed scheduling time - guaranteed CPU slices/memory etc (to prevent issues)
  • We use a EBS backend on these VMs if we make one more public then can we keep that backend so we can be sure we have a consistent envirionment? So that if we find an intermittent issue we can give the developers access to the box and they could get a box that is exactly the same as the one that saw the problem.
  • TODO: Look this up and make sure ^

Solutions

• embed with releng on the LWR project

TODO

• futurama meting with devs in toronto - set up
• set up game night thursday in toronto as well
• invite catlee to thursday
  • invite releng people to our futurama presentation we do for the entire team on the wednesday of our work week as well.

Clint's Monster Brainstorm List

• talos regression hunter automation
  • feed it a range a percentage above the trend line
  • and it bisects and outputs a set of changesets that test mostly below your trend line
• talos re-runner - specify a set of changes you can retest on talos
• orange quarantine
• VM on demand automatically sensing capacity filter
• Power consumption testing à la eideticker
  • Power tutor can help with this we had an intern working on this.
• Downloadable reproduction environment that simulates the automation style run in 5 clicks or less
• TBPL open JSON consumers - any automation anywhere can report to TBPL
• Test metrics - how good are the tests what do they cover, how redundant are they
• Syncrhonization mechanism between our tests and the w3c tests (bi-directional)
• One click bugzilla patch submission to try
• Better than github code reviews
• One click bugzilla checkin to land a patch
• Seamless minimal rooting kits, downloadable for developers
• stellar documentation on what to use when
• detangling mochitest from an "end all be all framework"
• Integration of QA automation systems with TBPL and their own reporting so that communication can happen between buildbot and devs for full release automation as well as uplifting of status.
• Mozharness try
• Networking labs - where we can simulate different networking loads for different types of tests
• Automation clusters where we provide seamless, extensible compute power and we move toward a more data-driven job scheduling model rather than the logic driven module now.
  • i.e. a slave reports in and says "I'm a winxp vm no hardware acceleration running 2gb of ram what can I do?" And it gets a job appropriate for it
• Benchmarks that actually measure things that matter and we can run them in automation as well as on websites for the world to see.
  • Memory intensive benchmarks
  • Performance benchmarks
  • Startup time benchmarks
  • Shutdown time benchmarks
  • Rendering over different network topologies
• Ability to expand automaton systems into those that have outside access and those that do not but provide seamless reporting across both.
• Tie crash reproducibility data from bughunter into soccoro
• Simple performance tests - no logic, just URLs
• One unified set of tools for writing test harnesses atop of - port all existing frameworks onto mozbase
• One unified manifest strategy for all tests - no makefiles
• Tie tests into unified front end for build system - mach, for instance
• Tie try reporting better into bugzilla - unified UI
• Tie datazilla and talos regression reporting into one unified dashboard TBPL or TBPL v2
• Allow anyone to merge and sheriff trees through automating the merge/backout process
• Reduce orange count by A) eliminating duplicate tests B) providing better reproducibility C) Fixing tests D) Fixing Code E) implementing the orange quarantine
• Moving all automation systems into one cluster for vms and machines alike - or perhaps two if we have to have some that should be separate for security reasons - things are too scattered right now
• Streamline deployment - create an automation computing system inside IT and truly staff and deliver it - manage the systems, flash the systems, repurpose systems, stock it with both hardware and VMs and network shapers etc. One large pool of workers and we can use them for various tasks through allocation APIs.
• Reduce turnaround time to 30minutes for try.
• Untangle the make file mess in the build system
• Make it simple to add a new set of tests to a framework.
• Make it possible to bisect over the tests in a framework to capture emergent issues that involve more than one test like memory leaks
• Make it possible to parallelize tests in extreme ways to take full advantage of the hardware. i.e. simulate keyboard and focus interactions so you cna run multiple applicaitons/test cycles per hardware instance.
• Bugmail is no longer an option. How do we ensure people see their bugs in a world where we are all seeing something insane - like the b2g bugmail area - it has something like 200 msgs/hr. How do we follow an issue through its lifecycle and know what needs to happen to it and who's on point? How do you queyr for that? How is that presented to you? Is there a bugzilla dashboard to end all dashbaords? Do we allow component watching with priority? How do we keep track of a bug changing states? i.e. from a "watching" to a "I've been cc'd" to a "someone needs smething from me" state. How can we highlight that. How can we see when that changes for one of our teammates so we can alert them or steal the review if we know they happen to be on vacation? Why can't we wire in backups and alternates into the review box? And have review suggestions based on module ownership?
• Are there mail-reader extensions we could do like a "secretary" that works like a junkmail filter in reverse and finds the things you like by analyzing the emails you linger over and the ones you skip over and have it be configurable enough so that you can have it start summarizing the days mail for you?
• How can we make profiling and debugging issues that only happen ith lots of iterations/data into tractable problems? How can we do debugging tools for truly intermittent problems? Can we extend existing tools?
• We should provide QA with simple tools and machines to build debug builds, run URLs until a crash is observed and then keep the session of what happened so a developer can replay the session and see the crash immediately in the debugger.
• There should be "bisect in the cloud" systems so that we can do mozregression searches using any defined test framework test as its criteria to find a good/bad build and it should hum along and produce a changeset for you where the build failed - these should exist for every platform - android to b2g to windows.
• One of the things people seem to love about github is the ability to quickly and easily do public tagging of bugs and then do a quick search by clicking the name from a tag cloud -- see the b2g gaia repo for an example of how this was used effectively. We could do something with components already. It might be useful for people working on an area to use these tags to further define their bugs - it could even simply be part of the whiteboard text but visiualized in a whole new way as a tag cloud.
• Scrumbugs integration by default w/o whiteboard tags.
• Better integration between bugzilla queries and the wiki
• Better ability to define standard burn-down metrics based on historical data in bugzilla. I suspect these already exist but no one knows how to create them so they build them on their own using the REST API
• ONE logging format.
• ONE logging parser, used as a module by everything else.
• Ability to really do test driven development - right now you can't because you can't checkin broken tests, and even if you tried this on try it would be worthless because you would wait too long for results. This would mean that each developer should be able to write their code and submit their test and run it on a particular set of platforms and get results back in 5 minutes or less. If you truly want to help people do TDD, then you need this infrastructure.
  • Revised to more like a "small try" if we could get builds fast/smart enough then you could do a build and a specific test and have that return in 5 minutes.
  • Could also do this with embedding people into their teams.
• We become part of the developer orientation for new developers so that they understand what the automation is and how to use it wisely from day one and they don't learn the bad habits and hearsay from the other developers.
• We never again have to explain a test framework/piece of automation by saying, "well it was originally built that way..." .... until two years from now.... ;)