Changes

# Smarter use of "Slimmer" memoryusage, e.g. more space-efficient data structures.# Avoiding leaks of all kinds. This includes both true leaks (where memory is lost forever) and "quasi-leaks" (where memory is not lost but held on for too long, e.g. due to infrequent garbage collection).

We get lots of leak reports. Many of them are vague. It can be hard to tell when duplicates happen. They often aren't closed. This means that tracking them at a coarse granulatiry (eg. the "mlk" keyword) ends up being of little use. So it's worthwhile tracking them on a per-release basis.

Primary The real goal: is to minimize bad performance (primarily paging) caused by poor memory usage. ThatBut that's hard to track, though.

A more trackable primary goal: get the number of leaks reported against a single version of Firefox down to zero, and keep it there. This will require some judgment on what constitutes a valid leak report, but that's no different to security bug reports, for example.

* To get in place reliable means The main automated infrastructure we currently have is the Tp4 and Tp5 Talos tests. They load a bunch of measuring popular (Alexa top 100) pages sequentially, take memory usage that detect when new leaks measurements (eg. private bytes on Windows) periodically, and other memory usage increases are introducedsomehow average those measurements into a single number. The current Tp4/Tp5 tracking is If that number increases significantly, certain people receive a start, but needs to be improvedwarning email.

Problems with this:* To get fix any leaks that existing tools (Valgrind, trace-malloc, etc) can find, and get infrastructure Sometimes increases in place the measurements fall through the cracks. I'm not sure what to prevent new ones being addeddo about this. The variation in measurements between runs doesn't help here.

* To get static analysis in place to automatically detect some kinds of leaksThey open only one tab at a time. This doesn't reflect how people browse, and doesn't stress the browser. In comparison, Sayre has ideas about this's [https://bugzilla.mozilla.org/show_bug.cgi?id=631705 Membuster] benchmark opens a heap of new tabs.

== areweslimyet* The "average of a sample taken every 20 seconds" is a curious measure.com ==

jmuizelaar set up [http://areweslimyet* The metrics are odd and inconsistent across platforms.com/ AWSY], inspired by [http Using the "explicit" and "resident" memory reporters (see about://arewefastyet.com AWFY]memory for details) would be better. It currently points to a [http://graphs-stage2.mozilla.org Perf-o-matic] Talos graph (Tp4 private bytes)Some statistics about page faults would also be good.

Why was AWFY so successful? Some key characteristics.* The benchmarks were easy to choose: everybody already used SS and V8, and then Mozilla pushed out Kraken. They run quickly, too, which is nice for devs, and allows expensive tools (like Cachegrind) to be used.* The metrics were easy to choose. SS time, V8 time, Kraken time. They can be measured easily, preciselyEven with these flaws, and fairly repeatably. You can break the total time down into per-benchmark times, which really helps with understanding improvements and regressions. Theythey're easy for devs to run on their own machine.* There was a well-defined goal: match or beat the other browsers.* Understanding where the time went was fairly easygood start, because but there are good time-oriented profilers: Shark, Cachegrind, etc's room to improve.

Talos benchmarks currently load a bunch * Sayre has some ideas about using static analysis in place to automatically detect some kinds of popular (Alexa top 100) pages sequentially, take measurements periodically, and somehow average those measurements into a single numberleaks.

Opening only one tab at * Telemetry will help a time is sillylot. Sayre's [httpsThere would be 2 kinds of telemetry://bugzillabuilt-in idle-daily stats and an addon.mozilla.org/show_bug Builtin idle-daily would report a sanitized version of about:memory, number of open tabs, among other non-privacy-invasive stats.cgi?id=631705 Membuster] benchmark looks promising; it opens We would also distribute a heap telemetry addon allowing users to opt into more invasive telemetry. This addon would be able to run benchmarks, and allow the users to explicitly trigger submission of new tabsdetailed privacy/perf-invasive information (such as open websites).

Repeatability * Mozmill endurance tests can track memory usage over time, which is a big issuemuch more informative than snapshot measurements or averaged measurements. If we load pages from the actual internetIt's unclear how much overhead they introduce, most of them will change frequentlyhowever. Using Leveraging them somehow could avoid a webpage recorder would be good (eg. [http://code.google.com/p/web-page-replay/ this one]), if it can be done. It might be worth redoing the recordings every 6 months to keep them up-to-date with the web's evolutionlot of duplicated work.

== Better Metrics areweslimyet.com ==

The following two metrics from aboutjmuizelaar set up [http:memory are the most useful//areweslimyet.com/ AWSY], inspired by [http://arewefastyet.com AWFY]. It currently points to a [http://graphs-stage2.mozilla.org Perf-o-matic] Talos graph (Tp4 private bytes).

Why was AWFY so successful? Some key characteristics.* "resident" is the amount of physical memory used. The benchmarks were easy to choose: It's the best measure of machine resources being everybody already used, but it's controlled by more-or-less opaque kernel page discard algorithmSS and V8, and depends on the behaviour of all processes in the system, which isn't greatthen Mozilla released Kraken. But if tests were always They run on the same machine and the browser was the only thing runningquickly, this might not be too bad. (No good for measuring once you get past peak, because jemalloc holds onto pages containing freed heap blocks? Not sure about that.) * "explicit" which is the memory explicitly requested by the browser, at the heap level (malloc, newnice for devs, etc) and the OS level allows expensive tools (mmap, VirtualAlloc, etclike Cachegrind). The heap-level component is accurate thanks to jemalloc; the OS-level component is an underestimate because it relies on having memory reporters in place. == A Clear Goal == Some ideas:* A certain percentage better than current, for whatever metric we choose. Arbitrary, but clear.* Within a certain percentage of other browsers (possibly including Firefox 3.6). Requires a metric that can be measured in other browsers easilyused.* No obvious leaks. Could be aimed for if we measure after closing lots of tabs.* Keep going until we can open N copies of TechCrunch on some standard phone? (I.e. until the mobile guys stop complaining)* Keep going until commenters on slashdot/Ars/reddit/etc stop complaining?? == A straw-man proposal == Just The metrics were easy to get discussion going..choose. === Benchmark === Something like Membuster: open a bunch of popular and/or interesting (eg. memory intensiveSS time, ones that caused problems in the past) pagesV8 time, one per tabKraken time. Not too many though; I suspect that opening 100 won't give us much more information that 10 would, and it should They can be runnable by devs measured easily.Use recorded versions for consistent results, if possibleprecisely, and make those recorded versions easy to use by devsfairly repeatably. ==== Mozmill tests ====Mozmill tests You can probably be used via break the Mozmill framework but there is no idea about the amount of overhead this introduces total time down into the browser. This, though, does make test writing trivialper-benchmark times, which really helps with understanding improvements and prevents duplication of the development of another frameworkregressions. === Metrics ===* Peak RSS: They're easy for devs to measure (run on all platforms -- I think?) across all browsers, has some reflection on actual performancetheir own machine.* Peak total: easy to measure across all browsers, interesting for the address space exhaustion issue. So, only two metrics per browser per run. Leaks aren't addressed; might be able to included them if we measure at startup and shutdown (after all tabs have closed). === Goal === Reduce peak RSS by 1/3. (Who knows? Probably need to measure other browsers to get a sense of what Understanding where the goal should be.)   == Other Ideas ===== User Reports ===Good metrics are more difficult for memory usage than they are for cpu time. Mostly this is badwent was fairly easy, but we may be able to use it to our advantage because it means it's harder for people to show that Firefox is using more memory than another browser. i.e. you can't load up the equivalent to sunspider in both browsers and immediately see which is better. This means that impact of bad memory usage is based on experience during regular browsing. I wonder if we could add something to Firefox input like "Report Firefox using too much memory" we could then upload the contents of about:memory. This can help us get a better picture of when our users have the perception that we're using too much memory. It also gives us a better opportunity to discover memory leaks that there are happening in the field. The above a approach kind of worked a few years back, with [[User:dbaron|dbaron]]'s memleak perl script and later extension. === Perf Team & Telemetry ===Telemetry {{bug|640791}} is currently blocked on getting a server. We plan to focus on memory stats as most important part of telemetry. There would be 2 kinds of telemetry: built-in idle-daily stats and an addon. Builtin idlegood time-daily would report a sanitized version of aboutoriented profilers:memory Shark, number of open tabsCachegrind, among other non-privacy-invasive statsetc. We would also distribute * There was a telemetry addon allowing users to opt into more invasive telemetry. This addon would be able to run benchmarks, and allow well-defined goal: match or beat the users to explicitly trigger submittion of detailed privacy/perf-invasive information(such as open websites)other browsers.

If we were focusing mostly on memory slimming a graph like AWFY's could work well. But given the primary goal above (zero valid leak reports against a future version of Firefox) we'd be better off having areweslimyet.com point [https://bugzilla.mozilla.org/showdependencytree.cgi?id=659860&hide_resolved== All hands notes ===We discussed this at the all hands in April 2011 the notes are 1 here: [[Performance/MemoryToolsTalk]].