CrashKill/Crashr
From MozillaWiki
Crash[r] -- Little snapshots of crash metrics on beta and final releases to help compare them.
this page has now been replaced with a charting tool built into http://crash-stats.mozilla.com/daily
here is what to play with
http://crash-stats.mozilla.com/daily?form_selection=by_version&p=Firefox&v[]=3.6&throttle[]=15&v[]=3.5.8&throttle[]=3.75&v[]=3.0.18&throttle[]=3.75&v[]=&throttle[]=100&os[]=Windows&os[]=Mac&os[]=Linux&date_start=2010-02-21&date_end=2010-03-07&submit=Generate
for 3.0.x and 3.5.x 25% of users opt in to sending (10% have box checked and another 15% check it) times 15% server throttle = 3.75% total effective throttle rate
for 3.6 its no client throttling and 15% server throttle since 1/29 full history of throttling adjustments for 3.6 is at https://wiki.mozilla.org/CrashKill/Crashr#3.6_RC1.2C_RC2.2C_Final
Contents
General Info
This is temporary repository for overall crash trend analysis until we can get better reporting systems in place.
- release dates from https://wiki.mozilla.org/Releases/
- adu's from metrics
- crash data from soccoro custom slices
- some releases are throttled. throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15)
- "williness to submit" follows what ever the defaults are (this assumption needs testing) early beta testers might be more willing to submit but we don't account for that complexity yet. maybe we can back out willness to submit after many release studies.
Here's the funny stuff that is missing.
Other observations and learning from the data
- crashes per hundred users should be ignored in the first couple days after release. while crash data is reported in real-time, ADU pings suffer from a delay of one or two days, which means that when there's a massive increase in usage in the day or two after a release, ADU data will be behind (crash data won't be) and the ratio will be way off.
- 3.0.14 was more stable than 3.5.3
- we started to get some more consistent daily measurements on 3.6b1 once we got 200-250k users
- Here is a picture of how throttling works, and some estimates on user populations of large and small releases.
- maybe 500k beta users, even throttled, is better than 200k unthrottled users in predicting stability; maybe confirming the theory that a bigger starting pool of users participating in the testing helps the numbers be more predictive.
- 200-250k users on 3.6b1 may have helped the numbers be much more consistent if not predictive (time will tell on the predictive part).
- we need to get 3.6b2 up to the level to try and do comparisons soon.
- having many users on the same release for at least a week or two helps week out outliers in daily or event fluctuations and gives time to make more sense of the data.
- on days when we have seen 250k throttled or raw adus of an unthrottled release the number of #crashes, #signatures, and crashes per 100 users are all still pretty noisy.
release throt adu date #crash #sig per/100 adj'ed 3.5.3 277241 18482710 20090911-crashdata.csv 34912 8799 0.19 12.59 3.0.14 221928 14795188 20090911-crashdata.csv 15634 4669 0.11 7.04 3.0.14 227750 15183311 20090912-crashdata.csv 28298 7328 0.19 12.43 3.6b1 -> 241000 20091109-crashdata.csv 16934 2694 7.03 <- 3.6b2 -> 200882 20091115-crashdata.csv 16223 2572 8.08 <- 3.6b2 -> 241726 20091116-crashdata.csv 16209 2627 6.71 <-
- The numbers in the 3.5.5 section https://wiki.mozilla.org/CrashKill/Crashr#Release_3.5.5 sugggest an interesting idea.
((47million_adus * 10%_client_throttle) * 15%_server_throttle)=700k users_willing&enabled-to submit crashes
if we had 700k beta testers we wouldn't have the diversity of users as the major 40million user release has, but for "common" crashes that everyone sees about equally we ought to see about the same volume. If these theories hold true that would remove one area of risk, and make comparisons a lot easier.
- we still may need to adjust willingness to submit in the case of throttling or larger user base, or seek a broader testing pool and more users participating. any ideas on how we might back out a "williness to submit" are welcome.
Release Snapshots
Release 3.6
Release 3.6b1
unthrottled
throttl adus date #crashes $sigs crashs/100users adjusted
25 200910132349 0 0 0.00
4000 200910142354 8 8 0.20
18000 200910152349 196 101 1.09
24000 200910162349 353 177 1.47
28000 200910172349 339 195 1.21
40000 200910182350 453 232 1.13
46000 200910192349 486 257 1.06
49000 200910202351 476 229 0.97
49000 200910212349 396 218 0.81
45000 200910222349 436 245 0.97
45000 200910232349 416 207 0.92
53000 200910242349 485 248 0.92
54000 200910252352 468 252 0.87
54000 200910262349 474 249 0.88
55000 200910272349 410 230 0.75
54000 200910282349 453 249 0.84
55000 200910292349 527 266 0.96
54000 200910302349 544 297 1.01
65000 200910312349 1049 418 1.61 b1 released
89000 200911012350 1468 500 1.65
123000 200911022349 1939 601 1.58
158000 200911032353 2177 645 1.38
180000 200911042352 2321 729 1.29
198500 200911052354 5962 1364 3.00
213000 200911062349 14917 2537 7.00
198000 200911072349 15187 2685 7.67
209000 200911082349 16679 2814 7.98
241000 200911092349 16934 2694 7.03
248000 200911102349 16092 2653 6.49 b2 released
231874 200911112354 11563 2222 4.99
147085 200911122350 6994 1561 4.76
109913 200911132354 5876 1417 5.35
87168 200911142354 5204 1265 5.97
78135 200911152353 4563 1164 5.84
83892 200911162354 4206 1155 5.01
77983 200911172354 3925 1067 5.03
73494 200911182353 3700 1108 5.03
71089 200911192354 3417 1018 4.81 b3 released
67131 200911202354 3160 912 4.71
61912 200911212354 3162 943 5.11
58483 200911222354 3162 988 5.41
67985 200911232354 3124 947 4.60
66186 200911242354 2949 952 4.46
63465 200911252354 2963 904 4.67
62903 200911262354 2780 877 4.42 b4 released
59665 200911272354 2687 872 4.50
54798 200911282354 2952 953 5.39
52991 200911292354 2818 947 5.32
61073 200911302354 2771 918 4.54
60188 200912012354 2894 944 4.81
59034 200912022354 3044 906 5.16
58093 200912032354 2915 853 5.02
Release 3.6b2
unthrottled
throttle adu's date #crashes #sigs crash/100users 200911072349 0 0 200911082349 0 0 169 200911092349 29 15 17.16 916 200911102349 773 229 84.39 27326 200911112354 7240 1323 26.49 128715 200911122350 10676 1847 8.29 171291 200911132354 13535 2273 7.90 177005 200911142354 14792 2445 8.36 200882 200911152353 16223 2572 8.08 241726 200911162354 16209 2627 6.71 257010 200911172354 15311 2787 5.96 253166 200911182353 10641 2277 4.20 143473 200911192354 5066 1437 3.53 84244 200911202354 3082 934 3.66 58037 200911212354 2473 859 4.26 45560 200911222354 1979 674 4.34 44188 200911232354 1884 651 4.26 36710 200911242354 1422 529 3.87 31747 200911252354 1272 479 4.01 28199 200911262354 1314 506 4.66 25474 200911272354 1286 462 5.05 23976 200911282354 1171 469 4.88 23312 200911292354 1128 454 4.84 25705 200911302354 1189 474 4.63 24856 200912012354 1176 424 4.73 23848 200912022354 1177 443 4.94 23111 200912032354 1247 403 5.40
Release 3.6b3
throt adu's date #crashes #sigs crash/100users
? 200911142354 0 0
? 200911152353 1 1
? 200911162354 18 8
? 200911172354 309 101
18435 200911182353 5028 1329 27.27
142847 200911192354 10331 2402 7.23
207349 200911202354 11423 2647 5.51
217975 200911212354 12906 2828 5.92
243541 200911222354 13796 3074 5.66
294307 200911232354 14574 3112 4.95
321004 200911242354 14927 3332 4.65
319230 200911252354 15826 3449 4.96
313303 200911262354 12941 3043 4.13
227788 200911272354 6720 1858 2.95
111492 200911282354 3578 1168 3.21
80372 200911292354 2727 970 3.39
79695 200911302354 2699 974 3.39
58951 200912012354 2138 746 3.63
47254 200912022354 1988 650 4.21
40100 200912032354 1574 560 3.93
Release 3.6b4
not throttled
throt adu's date #crashes #sigs crash/100users 200911242354 0 0 159 200911252354 37 24 23.27 11003 200911262354 3217 1100 29.24 -- release 101832 200911272354 11267 2984 11.06 208895 200911282354 14040 3561 6.72 262879 200911292354 16181 3992 6.16 318380 200911302354 16560 4035 5.20 354012 200912012354 17445 4230 4.93 377984 200912022354 18626 4437 4.93 394451 200912032354 18419 4482 4.67 399269 200912042353 18337 4398 4.59 375329 200912052354 18789 4588 5.01 390387 200912062354 19333 4668 4.95 447912 200912072354 20135 4824 4.50 460269 200912082354 20275 4980 4.41 472872 200912092353 20361 5003 4.31 479922 200912102353 20653 5033 4.30 464588 200912112354 20990 5248 4.52 446400 200912122354 20683 5165 4.63 459436 200912132354 21457 5397 4.67 529838 200912142354 22406 5425 4.23 533851 200912152354 22727 5729 4.26 539636 200912162354 21925 5448 4.06 536638 200912172354 21206 5421 3.95 482650 200912182354 13889 4006 2.88 244476 200912192354 7096 2586 2.90 161075 200912202354 4999 2040 3.10
Relases 3.6b5
adu's date #crashes #sigs crash/100users
1 200912042353 0 0
135 200912052354 3 3 2.22
556 200912062354 28 20 5.04
1199 200912072354 65 41 5.42
1874 200912082354 108 61 5.76
2361 200912092353 69 49 2.92
2864 200912102353 104 66 3.63
3223 200912112354 125 80 3.88
3187 200912122354 125 94 3.92
3607 200912132354 575 123 15.94
4746 200912142354 136 84 2.87
4994 200912152354 201 109 4.02
5122 200912162354 167 112 3.26
6113 200912172354 985 500 16.11 --release
50978 200912182354 7848 2604 15.39
259273 200912192354 14558 4145 5.61
357068 200912202354 16666 4526 4.67
n/a 200912212354 18467 5070 #DIV/0!
486443 200912222354 17982 4989 3.70
489788 200912232354 18820 4992 3.84
470922 200912242353 17752 4912 3.77
460922 200912252354 18097 4885 3.93
471404 200912262354 19242 5092 4.08
487620 200912272354 20527 5396 4.21
548337 200912282353 21708 5515 3.96
562748 200912292354 21374 5576 3.80
568013 200912302354 21922 5520 3.86
528415 200912312354 18786 5128 3.56
515642 201001012354 19605 5177 3.80
553745 201001022354 21457 5432 3.87
568976 201001032354 21586 5497 3.79
637244 201001042354 23540 5735 3.69
650022 201001052354 22468 5620 3.46
656689 201001062354 23275 5705 3.54
663466 201001072354 23204 5722 3.50
3.6 RC1, RC2, Final
adu's date #crashes #sigs crash/100users
4 201001042354 0 0 0.00
8 201001052354 0 0 0.00
678 201001062354 71 46 10.47
3093 201001072354 211 113 6.82
12593 201001082354 1758 749 13.96 -- RC1 released
42634 201001092354 5455 1878 12.79
156051 201001102354 10425 3156 6.68
303986 201001112354 15796 4277 5.20 -- RC1 announced
455997 201001122354 20082 5155 4.40
558918 201001132354 23174 5600 4.15
620807 201001142354 25135 5952 4.05
587824 201001152354 25220 5930 4.29
628645 201001162354 26040 6207 4.14 -- RC2 released
687117 201001172354 26920 6196 3.92
848320 201001182354 29956 6769 3.53 -- RC2 announced
835593 201001192354 30765 6868 3.68
866032 201001202354 31111 7027 3.59
trying some capacity experiments see https://bugzilla.mozilla.org/show_bug.cgi?id=541873
throt adu's date #crashes #sigs crash/100users adj'ed
169746 1131644 201001212354 11534 3841 1.02 6.79 release .15 throt.
819142 2340408 201001222354 38033 8617 1.63 4.64 50% throt 1/2day or 35%
1554227 3108455 201001232354 86637 14743 2.79 5.57 50% throt full day
1432037 4091536 201001242353 82166 14378 2.01 5.74 35% throt backed off to 25% 1/2 day
1348775 5395100 201001252354 59319 11927 1.10 4.4 25% throt all day
1545632 6182530 201001262353 70773 12315 1.14 4.58 25% throt all day
1734045 6936180 201001272354 79803 13514 1.15 4.6
1899947 7599789 201001282154 71366 13522 0.94 3.76 -socorro 1.4 downtime
1139956 7599708 201001292359 51680 11315 0.68 4.53 15% throt
1197119 7980796 201001302359 45635 10609 0.57 3.81
1116785 7445235 201001312359 47569 11062 0.64 4.26
2054602 8218407 201002012359 52568 11521 0.64 2.56
1501008 10006721 201002022359 57501 12230 0.57 3.83
1687279 11248526 201002032359 54787 11839 0.49 3.25
1762359 11749061 201002042359 60644 12972 0.52 3.44
1785989 11906591 201002051657 48740 11414 0.41 2.73
1630856 10872370 201002062359 61327 13090 0.56 3.76
1744223 11628156 201002072359 65058 13748 0.56 3.73
2096979 13979861 201002082359 71433 14431 0.51 3.41
3.6.4
adus day Tl-rpts num-sigs crash hang Crashes/100users
Reports/100users
20100401 462 287 462 0
20100402 681 351 681 0
20100403 690 428 690 0
20100404 767 417 767 0
20100405 726 373 726 0
20100406 741 383 741 0
20100407 843 415 843 0
20100408 739 392 739 0
20100409 707 385 707 0
20100410 737 411 737 0
20100411 580 324 580 0
20100412 705 383 705 0
20100413 768 366 768 0
305 20100414 790 395 790 0 259.02 259.02
692 20100415 787 407 787 0 113.73 113.73
9794 20100416 1196 540 1196 0 12.21 12.21
85949 20100418 9248 2539 9248 0 10.76 10.76
160152 20100419 11871 3104 11871 0 7.41 7.41
233291 20100420 12392 3221 12392 0 5.31 5.3
270527 20100421 16781 4322 16781 0 6.2 6.2
302005 20100422 16043 3947 16043 0 5.31 5.31
310027 20100423 18156 4371 18156 0 5.86 5.86
281726 20100424 15923 3907 15923 0 5.65 5.65
297872 20100425 15844 3921 15844 0 5.32 5.32
361567 20100426 15283 3777 15283 0 4.23 4.23
370104 20100427 15153 3734 15153 0 4.09 4.09
367489 20100428 15662 3873 12658 3004 3.44 4.26 <socorro1.6>
367367 20100429 14967 3600 4156 10811 1.13 4.07
360720 20100430 15855 3808 4165 11690 1.15 4.4
311825 20100501 15628 3786 4083 11545 1.31 5.01
329985 20100502 15611 3851 4398 11213 1.33 4.73
382206 20100503 15392 3905 5232 10160 1.37 4.03
400797 20100504 14868 3767 5560 9308 1.39 3.71
405217 20100505 15784 3707 6128 9656 1.51 3.9
412228 20100506 17807 3852 7957 9850 1.93 4.32
406629 20100507 19322 4005 9275 10047 2.28 4.75
359375 20100508 19898 3260 9738 10160 2.71 5.54
365403 20100509 19742 3162 9812 9930 2.69 5.4
436228 20100510 19971 3121 9942 10029 2.28 4.58
446153 20100511 21107 3191 10554 10553 2.37 4.73
449961 20100512 20166 2938 10816 9350 2.4 4.48
440549 20100513 21073 3167 10618 10455 2.41 4.78
442143 20100514 20931 3056 10788 10143 2.44 4.73
389488 20100515 20097 3104 10378 9719 2.66 5.16
397634 20100516 18824 3230 8568 10256 2.15 4.73
473572 20100517 17933 3162 8052 9881 1.7 3.79
473517 20100518 17922 3198 7540 10382 1.59 3.78
3.6.4 redo
the socorro 1.6 update allows a bit more fine grained analysis to breakdown where the crashes are happening. from the .csv reports we can now determine:
* tl_rpts are all the reports we get
* crashes
fx_crsh = process_type ~ \N && hangid ~ \N
p_crsh = process_type ~ [plugin] && hangid ~ \N
* hang pairs
fx_hang = process_type ~ \N && hangid !~ \N
p_hang = process_type ~ [plugin] && hangid !~ \N
* tl_events subtract out about half of the hang pair report (just uses fx_hang)
* fx is the pct. of tl_events that are firefox crashes (browser restart required)
* oopp is the pct. of tl_events that are out or process (page reload/plugin restart required)
date tl_rpts fx_crsh p_crsh [hang pair] oopp_event tl_events fx oopp oopcrsh oophang
fx plugin
20100512 20169 2907 5504 [6402 5356] 11906 13767 0.211 0.865 0.400 0.465
20100513 21075 2452 6818 [5873 5932] 12691 15202 0.161 0.835 0.448 0.386
20100514 20931 2561 6867 [5709 5794] 12576 15222 0.168 0.826 0.451 0.375
20100515 20098 2689 6444 [5437 5528] 11881 14661 0.183 0.810 0.440 0.371
20100516 18826 2533 4714 [5767 5812] 10481 13059 0.194 0.803 0.361 0.442
20100517 17934 2626 4138 [5562 5608] 9700 12372 0.212 0.784 0.334 0.450
20100518 17939 2650 3642 [5800 5847] 9442 12139 0.218 0.778 0.300 0.478
20100519 19540 2679 3333 [6738 6790] 10071 12802 0.209 0.787 0.260 0.526
more data now at
http://people.mozilla.com/~chofmann/crash-stats/ date / crash-counts.txt
Other Branch Comparisons
Release 3.0.16
Assumes:
- 1 in 10 keep default opt-in
- 1.5 in 10 override default opt-out and participate in sending
- 7.5 keep default opt out
throt adu's date #crashes #sigs crashes/100users adj'ed 690934 18424903 200912202354 31573 8839 0.17 4.57 882320 23528535 200912212354 33796 9350 0.14 3.83 900637 24016981 200912222354 32934 9315 0.14 3.66 878647 23430588 200912232354 31838 8980 0.14 3.62 750945 20025201 200912242353 27882 8178 0.14 3.71 684988 18266357 200912252354 27312 7937 0.15 3.99 729035 19440943 200912262354 29740 8433 0.15 4.08 761571 20308573 200912272354 32072 8922 0.16 4.21 914821 24395238 200912282353 34889 9567 0.14 3.81 918464 24492361 200912292354 34346 9626 0.14 3.74 903389 24090382 200912302354 33410 9127 0.14 3.70 756432 20171530 200912312354 26586 7795 0.13 3.51 741078 19762079 201001012354 28990 8310 0.15 3.91 802441 21398426 201001022354 31661 8891 0.15 3.95 837543 22334477 201001032354 33591 9333 0.15 4.01
Release 3.5.6
Assumes:
- 1 in 10 keep default opt-in
- 1.5 in 10 override default opt-out and participate in sending
- 7.5 keep default opt out
throt adu's date #crashes #sigs crash/100users adj'ed 0 1 200911032353 1 1 100.00 2666.67 4 108 200911042352 3 3 2.78 74.07 26 686 200911052354 8 5 1.17 31.10 39 1050 200911062349 21 16 2.00 53.33 127 3392 200911072349 59 37 1.74 46.38 196 5219 200911082349 40 26 0.77 20.44 257 6845 200911092349 22 14 0.32 8.57 329 8777 200911102349 34 24 0.39 10.33 367 9797 200911112354 21 17 0.21 5.72 392 10458 200911122350 26 20 0.25 6.63 423 11270 200911132354 51 33 0.45 12.07 413 11012 200911142354 29 19 0.26 7.02 348 9274 200911152353 21 17 0.23 6.04 450 12001 200911162354 47 33 0.39 10.44 466 12424 200911172354 32 26 0.26 6.87 479 12780 200911182353 31 27 0.24 6.47 479 12772 200911192354 27 21 0.21 5.64 464 12367 200911202354 23 18 0.19 4.96 387 10315 200911212354 34 18 0.33 8.79 400 10671 200911222354 36 19 0.34 9.00 497 13258 200911232354 38 32 0.29 7.64 501 13371 200911242354 32 21 0.24 6.38 489 13033 200911252354 25 20 0.19 5.12 416 11094 200911262354 20 14 0.18 4.81 518 13800 200911272354 24 22 0.17 4.64 516 13765 200911282354 28 28 0.20 5.42 2003915 11094 200911292354 29 29 0.26 0.00 332 8866 200911302354 34 30 0.38 10.23 286 7626 200912012354 36 29 0.47 12.59 8 215 200912022354 2 2 0.93 24.81 23 621 200912032354 2 2 0.32 8.59 59 1580 200912042353 41 38 2.59 69.20 1435 38261 200912052354 424 311 1.11 29.55 7166 191092 200912062354 740 519 0.39 10.33 10051 268018 200912072354 845 568 0.32 8.41 12655 337479 200912082354 932 624 0.28 7.36 13648 363938 200912092353 967 665 0.27 7.09 10376 276706 200912102353 983 629 0.36 9.47 13583 362226 200912112354 953 612 0.26 7.02 12320 328527 200912122354 963 636 0.29 7.82 12630 336812 200912132354 1033 671 0.31 8.18 15283 407549 200912142354 1072 684 0.26 7.01 15382 410196 200912152354 3071 1603 0.75 19.96 70961 1892297 200912162354 38537 10288 2.04 54.31 --- release 1001744 26713178 200912172354 80443 16951 0.30 8.03 1408636 37563622 200912182354 94168 18885 0.25 6.69 1396724 37245967 200912192354 97103 19127 0.26 6.95 1547848 41275945 200912202354 105852 19938 0.26 6.84 1881333 50168875 200912212354 113209 21145 0.23 6.02 1930619 51483178 200912222354 111865 21156 0.22 5.79 1902380 50730145 200912232354 108806 20795 0.21 5.72 1685839 44955701 200912242353 95784 18990 0.21 5.68 1578158 42084225 200912252354 93840 18579 0.22 5.95 1661335 44302279 200912262354 103336 19964 0.23 6.22 1738281 46354170 200912272354 110489 20633 0.24 6.36 2003915 53437740 200912282353 118482 22105 0.22 5.91 2027493 54066489 200912292354 117276 21892 0.22 5.78 2012969 53679184 200912302354 115481 21262 0.22 5.74 1743204 46485440 200912312354 93512 18576 0.20 5.36 1744223 46512624 201001012354 100126 19477 0.22 5.74 1856723 49512624 201001022354 109588 20749 0.22 5.90 1938716 51699096 201001032354 114935 21535 0.22 5.93
Older Releases
Release 3.5.5
throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15)
throt adus date #crashes #sigs crash/100users throttle adj'ed 0 200911012350 0 0 0 200911022349 1 1 2 138 200911032353 0 0 0.00 0.00 4 295 200911042352 3 3 1.02 67.80 129 8588 200911052354 3548 1580 41.31 2754.23 -releasee 33912 2260795 200911062349 41899 9698 1.85 123.55 321576 21438378 200911072349 81941 15360 0.38 25.48 455219 30347934 200911082349 95888 17386 0.32 21.06 589113 39274168 200911092349 100690 18062 0.26 17.09 657614 43840924 200911102349 104768 18557 0.24 15.93 684712 45647479 200911112354 110074 19201 0.24 16.08 714202 47613465 200911122350 100894 18127 0.21 14.13 704354 46956936 200911132354 107604 18872 0.23 15.28 619366 41291046 200911142354 104979 18519 0.25 16.95 654919 43661269 200911152353 111029 19122 0.25 16.95 773046 51536376 200911162354 115214 19656 0.22 14.90 786625 52441671 200911172354 124840 20048 0.24 15.87 793275 52885030 200911182353 119438 19943 0.23 15.06 793123 52874870 200911192354 115156 19858 0.22 14.52 767605 51173636 200911202354 112276 19719 0.22 14.63 672077 44805118 200911212354 109937 19335 0.25 16.36 699671 46644737 200911222354 117899 20390 0.25 16.85 826446 55096414 200911232354 120234 20916 0.22 14.55
Release 3.5.4
throttle assumes 10% client throttling, then 15% server processing or ((adus * .10) * .15)
throt adu's date #crashes #sigs crash/100users adj'ed 5975 398311 20091013-crashdata.csv 1457 793 0.37 24.39 6575 438328 20091014-crashdata.csv 1718 900 0.39 26.13 6815 454324 20091015-crashdata.csv 1597 780 0.35 23.43 6795 452985 20091016-crashdata.csv 1635 809 0.36 24.06 5932 395453 20091017-crashdata.csv 1653 843 0.42 27.87 6133 408839 20091018-crashdata.csv 1706 867 0.42 27.82 7360 490677 20091019-crashdata.csv 1819 896 0.37 24.71 7492 499473 20091020-crashdata.csv 1747 910 0.35 23.32 7533 502184 20091021-crashdata.csv 1695 905 0.34 22.5 7512 500774 20091022-crashdata.csv 1740 876 0.35 23.16 7275 484989 20091023-crashdata.csv 1795 912 0.37 24.67 6334 422238 20091024-crashdata.csv 1738 906 0.41 27.44 6515 434366 20091025-crashdata.csv 1852 908 0.43 28.42 7680 511986 20091026-crashdata.csv 1730 895 0.34 22.53 7730 515329 20091027-crashdata.csv 2825 1310 0.55 36.55 27357 1823786 20091028-crashdata.csv 34007 8190 1.86 124.31 -release 344243 22949509 20091029-crashdata.csv 79135 15053 0.34 22.99 488519 32567919 20091030-crashdata.csv 92431 16607 0.28 18.92 534332 35622134 200911012350 105485 18390 0.3 19.74 655674 43711622 200911022349 108838 18666 0.25 16.6 692718 46181179 200911032353 110940 18843 0.24 16.02 702750 46850020 200911042352 107798 18628 0.23 15.34 706014 47067594 200911052354 104667 18213 0.22 14.83 -3.5.5repl 668776 44585086 200911062349 69825 13500 0.16 10.44 286885 19125665 200911072349 31838 7655 0.17 11.1 186887 12459142 200911082349 21151 5723 0.17 11.32 171256 11417053 200911092349 16872 4860 0.15 9.85 114661 7644079 200911102349 12372 3824 0.16 10.79 86634 5775579 200911112354 10099 3290 0.17 11.66 67889 4525902 200911122350 7938 2692 0.18 11.69 53840 3589364 200911132354 7426 2474 0.21 13.79 41540 2769331 200911142354 6667 2321 0.24 16.05 37504 2500244 200911152353 6019 2181 0.24 16.05 40472 2698155 200911162354 5937 2135 0.22 14.67 35387 2359153 200911172354 5445 1918 0.23 15.39 31936 2129046 200911182353 5098 1810 0.24 15.96 28871 1924766 200911192354 4830 1745 0.25 16.73 25825 1721658 200911202354 4637 1688 0.27 17.96 25825 1721658 200911212354 4364 1605 0.25 16.9 21167 1411151 200911222354 3993 1526 0.28 18.86 20086 1339089 200911232354 4113 1557 0.31 20.48 23053 1536836 200911242354 3856 1534 0.25 16.73 21460 1430689 200911252354 3734 1436 0.26 17.4 19610 1307319 200911262354 3392 1276 0.26 17.3 17672 1178106 200911272354 3586 1292 0.3 20.29 16183 1078880 200911282354 3185 1158 0.3 19.68 13890 926007 200911292354 3123 1179 0.34 22.48 13751 916761 200911302354 3269 1194 0.36 23.77
An example of extreme growth in a regression crash causing a respin of 3.5.4
https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20status1.9.1%3A.5-fixed
6 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091020-crashdata 5 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091021-crashdata 4 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091022-crashdata 9 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091023-crashdata 14 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091024-crashdata 10 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091025-crashdata 9 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091026-crashdata 33 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091027-crashdata 3.5.4 released -- 506 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091028-crashdata 1034 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091029-crashdata 712 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091030-crashdata 556 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091031-crashdata 597 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091101-crashdata 540 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091102-crashdata 474 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091103-crashdata 404 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091104-crashdata 371 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091105-crashdata 249 crashes for gfxWindowsFontGroup::WhichFontSupportsChar on 20091106-crashdata
On 20091024 just a few days before shipping that signature ranked #85 with 3 crashes that day.
85. 3 gfxWindowsFontGroup::WhichFontSupportsChar(nsTArray<nsRefPtr<FontEntry> > const&, unsigned int) 3.5.4
20091029 ranking
2. 1029 gfxWindowsFontGroup::WhichFontSupportsChar(nsTArray<nsRefPtr<FontEntry> > const&, unsigned int) 3.5.4
Release 3.5.3
throt adu's date #crashes #sigs crash/100users adj'ed 22 1466 20090901-crashdata.csv 7 5 0.48 31.83 719 47955 20090902-crashdata.csv 52 46 0.11 7.23 4959 330590 20090903-crashdata.csv 744 492 0.23 15 6180 411973 20090904-crashdata.csv 1319 734 0.32 21.34 5769 384603 20090905-crashdata.csv 1584 856 0.41 27.46 6151 410099 20090906-crashdata.csv 1444 802 0.35 23.47 7407 493812 20090907-crashdata.csv 1534 789 0.31 20.71 7965 530986 20090908-crashdata.csv 1625 892 0.31 20.4 8476 565087 20090909-crashdata.csv 1703 937 0.3 20.09 -- release 44559 2970600 20090910-crashdata.csv 5219 2081 0.18 11.71 277241 18482710 20090911-crashdata.csv 34912 8799 0.19 12.59 291479 19431957 20090912-crashdata.csv 61423 13297 0.32 21.07 382936 25529036 20090913-crashdata.csv 72537 14862 0.28 18.94 505111 33674054 20090914-crashdata.csv 87536 17086 0.26 17.33 553302 36886822 20090915-crashdata.csv 95342 18220 0.26 17.23 575617 38374498 20090916-crashdata.csv 97072 18528 0.25 16.86 588830 39255365 20090917-crashdata.csv 98769 18508 0.25 16.77 574876 38325034 20090918-crashdata.csv 98938 18718 0.26 17.21 490791 32719389 20090919-crashdata.csv 97003 18655 0.3 19.76 525336 35022427 20090920-crashdata.csv 94472 18103 0.27 17.98 621540 41436022 20090921-crashdata.csv 102442 19071 0.25 16.48 636066 42404407 20090922-crashdata.csv 106806 19752 0.25 16.79 645549 43036631 20090923-crashdata.csv 106061 19523 0.25 16.43 648983 43265543 20090924-crashdata.csv 106873 19798 0.25 16.47 628283 41885531 20090925-crashdata.csv 105275 19760 0.25 16.76 541861 36124056 20090926-crashdata.csv 109329 19166 0.3 20.18 570765 38051029 20090927-crashdata.csv 110597 19062 0.29 19.38 676617 45107781 20090928-crashdata.csv 108851 19565 0.24 16.09 684282 45618770 20090929-crashdata.csv 110626 20073 0.24 16.17 18.27
Release 3.0.x
Release 3.0.14
throt adu's date #crashes #sigs crash/100users adj'ed 4 299 20090901-crashdata.csv 1 1 0.33 22.30 133 8881 20090902-crashdata.csv 10 9 0.11 7.51 1113 74179 20090903-crashdata.csv 111 92 0.15 9.98 1435 95637 20090904-crashdata.csv 154 123 0.16 10.74 1329 88567 20090905-crashdata.csv 211 155 0.24 15.88 1441 96037 20090906-crashdata.csv 198 150 0.21 13.74 1853 123525 20090907-crashdata.csv 195 160 0.16 10.52 1967 131110 20090908-crashdata.csv 197 150 0.15 10.02 2062 137469 20090909-crashdata.csv 224 184 0.16 10.86 - release 24712 1647489 20090910-crashdata.csv 1405 741 0.09 5.69 221928 14795188 20090911-crashdata.csv 15634 4669 0.11 7.04 227750 15183311 20090912-crashdata.csv 28298 7328 0.19 12.43 311923 20794866 20090913-crashdata.csv 32374 8636 0.16 10.38 432973 28864876 20090914-crashdata.csv 41727 10312 0.14 9.64 480558 32037233 20090915-crashdata.csv 46357 10951 0.14 9.65 500022 33334831 20090916-crashdata.csv 47023 11157 0.14 9.40 510106 34007050 20090917-crashdata.csv 47409 11296 0.14 9.29 490892 32726129 20090918-crashdata.csv 47116 11218 0.14 9.60 406300 27086687 20090919-crashdata.csv 45673 11156 0.17 11.24 439670 29311359 20090920-crashdata.csv 43943 10913 0.15 9.99 535907 35727115 20090921-crashdata.csv 48341 11631 0.14 9.02 544515 36301033 20090922-crashdata.csv 49944 11692 0.14 9.17 549171 36611388 20090923-crashdata.csv 49985 11796 0.14 9.10 548170 36544655 20090924-crashdata.csv 49626 11699 0.14 9.05 524175 34945024 20090925-crashdata.csv 48913 11522 0.14 9.33 438887 29259151 20090926-crashdata.csv 58846 11362 0.2 13.41 465577 31038440 20090927-crashdata.csv 61053 11084 0.2 13.11 569360 37957328 20090928-crashdata.csv 53189 11947 0.14 9.34 570608 38040564 20090929-crashdata.csv 52586 12059 0.14 9.22 10.57
