QA Feature Specification:Automating Timezone Tests

These thoughts were compiled and emailed in response to Ray's post in bug 369270 comment 8. These are some very general, first pass thoughts on how to automate timezone testing. Please feel free to add your own.

I would love to have a way to automate testing of timezones. There are two levels of timezone testing that we would need to automate. The first is the low-level C++ timezone interfaces that we use (the ones calling directly into libical). I think the easiest way to test these would be to write a few unit test style xpcshell tests. Indeed, I plan to work on these next week in order to prepare the road for refactoring much of our timezone backend implementation. I'll need those unit tests to be certain that I don't break anything. Honestly, these unit tests *should* already exist on the product, and I don't think of them as true test automation in my mind (from a QA standpoint).

The second level of the timezone testing is the part that I would consider worth a full QA based automation strategy. This is the view part. Basically, the problem is that there are a lot of objects that interact with dates from their journey out of a provider, in and through libical, through the MVC levels and into boxes and UI that the user sees. All along the way, date calculations are being made and decided. So, I think we need a "view-level" automation. We have the beginnings of this effort in an extension that Jminta built: bug 358985.

I think that the basic methodology for an automated timezone test would be something like the following:

* for each calendar provider:
* for each timezone...
   * create a "standard set" of items (sequential and recurring - see below) in timezone[i]
   * for each timezone...
      * Change view to timezone[j]
      * Verify that the correct date and time is displayed for these events when viewed in timezone[j]

That's obviously exceedingly general and exceedingly inefficient. But, something like this that does an exhaustive test of all timezones via viewing those timezones through the lens of all other timezones would be a great automated test. Another method would be to generate a massive set of items such that you have a "standard set" of items in each timezone on the list. Then you would loop through all the timezones, changing the view to timezone[i] and verify that each item is displayed on the proper date and time in the view on timezone[i]. (essentially, the only loop you would need would be the inner loop from the above algorithm).

The Standard Set of Items

This "standard set" of items will satisfy all these requirements:

• You need each type of recurring appointment to be represented
• You need non recurring items
• You need both all day and date-time style items to be represented
• You need both tasks and events (hence why I keep saying "item") represented
• You need to have items in what I have come to call "danger" hour blocks as well as "normal" hour blocks on each zone:
  • This means that you need to have items in your set that have the following constraints (t = datetime of item, o = offset from UTC of time zone):
    • t <=abs( o) + 00:00 -- t is within the timezone offset from midnight in the AM (used to test day rollover)
    • t >= 23:59 - abs(o) -- t is within the timezone offset from midnight in the PM (used to test day rollover)
    • o + 00:00 <= t <= 23:59 - o -- t is between the "dangerous" offset times above.
    • An example: I want to test the UTC - 6 timezone, so I need to ensure I have at least one item in each of these following time ranges: 00:00 to 06:00, 06:00 to 18:00, 18:00 to 23:59.

Verification Issues

When verifying, you need to verify in the following sections of the year:

• Outside DST, prior to DST change for that zone, for that year
• Verify items on the two days leading up to going into DST change (to be sure our starting DST date is correct)
• Verify items on the two days following DST change (to be sure our starting DST date is correct)
• Inside DST
• Verify items on the two days prior to shifting out of DST (to be sure our ending of DST date is correct)
• Verify items on the two days after shifting out of DST (to be sure our ending of DST date is correct)
• Outside DST, after DST change for that zone, for that year (if exists for that zone).

Note that there is a marked difference in the DST cycle for northern hemisphere and southern hemisphere zones when considered in any given year:

Northern zone looks like this:
J   F   M   A   M   J   J   A   S   O   N   D
-noDst------|---------Dst-----------|---noDST--
Area A     Bgn     Area B       End Area C

Southern zone is like this:
J   F   M   A   M   J   J   A   S   O   N   D
-Dst------|---------NoDst-----------|---DST--
Area B   End         Area A        Bgn  Area B

Where:

• Area A - before beginning of DST for the year
• Area B - in DST
• Area C - after ending of DST for the year
• Bgn - marks the beginning of DST in the year
• End - marks the ending of DST in the year

User interaction automation with items in these zones

I don't know how automated a test you are imagining, but it would be great to have the ability to also test the following on a subset of the items in a given year:

• Moving a recurring item (full sequence edit)
• Moving a recurring item (instance edit)
• Ensure proper time is displayed when edit event/task window is loaded
• Editing an event does not change timezone setting.