Confirm
1,345
edits
Changes
Created page with "MemShrink aims to reduce Firefox's memory consumption. There are three potential benefits: # Speed: less cache and VM pressure. # Stability: fewer OOMs, whether due to addres..."
MemShrink aims to reduce Firefox's memory consumption. There are three potential benefits:
# Speed: less cache and VM pressure.
# Stability: fewer OOMs, whether due to address space exhaustion or otherwise. This results in fewer crashes (due to mishandling of OOM) or aborts.
# Perception: fewer people will complain about Firefox being a memory hog.
== areweslimyet.com ==
jmuizelaar set up [http://areweslimyet.com/ AWSY], inspired by [http://arewefastyet.com AWFY]. It currently points to a [http://graphs-stage2.mozilla.org Perf-o-matic] Talos graph (Tp4 private bytes).
Why was AWFY so successful? Some key characteristics.
* The benchmarks were easy to choose: everybody already used SS and V8, and then Mozilla pushed out Kraken. They run quickly, too, which is nice for devs, and allows expensive tools (like Cachegrind) to be used.
* The metrics were easy to choose. SS time, V8 time, Kraken time. They can be measured easily, precisely, and fairly repeatably. You can break the total time down into per-benchmark times, which really helps with understanding improvements and regressions. They're easy for devs to run on their own machine.
* There was a well-defined goal: match or beat the other browsers.
* Understanding where the time went was fairly easy, because there are good time-oriented profilers: Shark, Cachegrind, etc.
In comparison, for AWSY there is no good and/or standard benchmark suite, metrics are less clear and/or harder to measure, there's no well-defined goal ("better than it currently is" isn't enough), and the memory profiling tools aren't very good. Let's consider each of those in turns.
== Better Benchmarks ==
Talos benchmarks currently load a bunch of popular (Alexa top 100) pages sequentially, take measurements periodically, and somehow average those measurements into a single number.
Opening only one tab at a time is silly. Sayre's [https://bugzilla.mozilla.org/show_bug.cgi?id=631705 Membuster] benchmark looks promising; it opens a heap of new tabs.
Repeatability is a big issue. If we load pages from the actual internet, most of them will change frequently. Using a webpage recorder would be good, if it can be done. It might be worth redoing the recordings every 6 months to keep them up-to-date with the web's evolution.
== Better Metrics ==
Picking good metrics is difficult, especially if we want to compare against other browsers.
* Total memory is only moderately relevant, because lots of mapped pages aren't touched and so don't cost anything. (Thread stacks are a good example -- they're 8MB each on 64-bit Linux.) The exception is on 32-bit machines where address space exhaustion is possible.
* RSS (resident set size) is more relevant from a performance point of view. But it's controlled by some more-or-less opaque kernel page discard algorithm, and depends on the behaviour of all processes in the system, which isn't great. But if tests were always run on the same machine and the browser was the only thing running, this might not be too bad. (No good for measuring once you get past peak, because jemalloc holds onto pages containing freed heap blocks? Not sure about that.)
* We could do something like the size of all heap allocations plus other notable non-heap allocations (like code pages allocated by the JITs). This requires some internal accounting and so is hard to compare against other browsers.
* What do we do about multiple processes? Measure them all and sum the numbers?
* There seems to be a lot of non-determinism in browser memory usage, which makes pinpointing small regressions/improvements difficult.
* Do we only measure peak usage? Or do we want to open and close lots of tabs to try to catch leaks?
* jseward's [http://blog.mozilla.com/jseward/2011/01/27/profiling-the-browsers-virtual-memory-behaviour/ VM-sim extension to Cachegrind] takes a different tack; it measures page faults on a quasi-idealized machine. It's slow, though.
== A Clear Goal ==
Some ideas:
* A certain percentage better than current, for whatever metric we choose. Arbitrary, but clear.
* Within a certain percentage of other browsers (possibly including Firefox 3.6). Requires a metric that can be measured in other browsers easily.
* No obvious leaks. Could be aimed for if we measure after closing lots of tabs.
* Keep going until we can open N copies of TechCrunch on some standard phone? (I.e. until the mobile guys stop complaining)
* Keep going until commenters on slashdot/Ars/reddit/etc stop complaining??
== Better Profiling ==
nnethercote has plans for greatly improving about:memory. See [http://blog.mozilla.com/nnethercote/2011/02/09/a-vision-for-better-memory-profiling-with-aboutmemory/ this blog post] for details. These will make it much easier for everyone to understand how Firefox is using memory.
== A Straw-man Proposal ==
Just to get discussion going...
=== Benchmark ===
Something like Membuster: open a bunch of popular and/or interesting (eg. memory intensive, ones that caused problems in the past) pages, one per tab.
Not too many though; I suspect that opening 100 won't give us much more information that 10 would, and it should be runnable by devs easily.
Use recorded versions for consistent results, if possible, and make those recorded versions easy to use by devs.
=== Metrics ===
* Peak RSS: easy to measure (on all platforms -- I think?) across all browsers, has some reflection on actual performance.
* Peak total: easy to measure across all browsers, interesting for the address space exhaustion issue.
So, only two metrics per browser per run. Leaks aren't addressed; might be able to included them if we measure at startup and shutdown (after all tabs have closed).
=== Goal ===
Reduce peak RSS by 1/3.
(Who knows? Probably need to measure other browsers to get a sense of what the goal should be.)
== Pushing This Forward ==
nnethercote and jmuizelaar have volunteered to co-lead the MemShrink effort.
MemShrink could be modelled on CrashKill and CritSmash. One problem with that is that neither of them are in Mountain View -- jmuizelaar is in the Toronto office (not too bad), nnethercote works at home in a PST+17/+19 timezone. So having CrashKill/CritSmash-style meetings will be difficult.
The MemShrink effort can also include leaks and quasi-leaks in its jurisdiction, even if areweslimyet.com doesn't demonstrate them.
# Speed: less cache and VM pressure.
# Stability: fewer OOMs, whether due to address space exhaustion or otherwise. This results in fewer crashes (due to mishandling of OOM) or aborts.
# Perception: fewer people will complain about Firefox being a memory hog.
== areweslimyet.com ==
jmuizelaar set up [http://areweslimyet.com/ AWSY], inspired by [http://arewefastyet.com AWFY]. It currently points to a [http://graphs-stage2.mozilla.org Perf-o-matic] Talos graph (Tp4 private bytes).
Why was AWFY so successful? Some key characteristics.
* The benchmarks were easy to choose: everybody already used SS and V8, and then Mozilla pushed out Kraken. They run quickly, too, which is nice for devs, and allows expensive tools (like Cachegrind) to be used.
* The metrics were easy to choose. SS time, V8 time, Kraken time. They can be measured easily, precisely, and fairly repeatably. You can break the total time down into per-benchmark times, which really helps with understanding improvements and regressions. They're easy for devs to run on their own machine.
* There was a well-defined goal: match or beat the other browsers.
* Understanding where the time went was fairly easy, because there are good time-oriented profilers: Shark, Cachegrind, etc.
In comparison, for AWSY there is no good and/or standard benchmark suite, metrics are less clear and/or harder to measure, there's no well-defined goal ("better than it currently is" isn't enough), and the memory profiling tools aren't very good. Let's consider each of those in turns.
== Better Benchmarks ==
Talos benchmarks currently load a bunch of popular (Alexa top 100) pages sequentially, take measurements periodically, and somehow average those measurements into a single number.
Opening only one tab at a time is silly. Sayre's [https://bugzilla.mozilla.org/show_bug.cgi?id=631705 Membuster] benchmark looks promising; it opens a heap of new tabs.
Repeatability is a big issue. If we load pages from the actual internet, most of them will change frequently. Using a webpage recorder would be good, if it can be done. It might be worth redoing the recordings every 6 months to keep them up-to-date with the web's evolution.
== Better Metrics ==
Picking good metrics is difficult, especially if we want to compare against other browsers.
* Total memory is only moderately relevant, because lots of mapped pages aren't touched and so don't cost anything. (Thread stacks are a good example -- they're 8MB each on 64-bit Linux.) The exception is on 32-bit machines where address space exhaustion is possible.
* RSS (resident set size) is more relevant from a performance point of view. But it's controlled by some more-or-less opaque kernel page discard algorithm, and depends on the behaviour of all processes in the system, which isn't great. But if tests were always run on the same machine and the browser was the only thing running, this might not be too bad. (No good for measuring once you get past peak, because jemalloc holds onto pages containing freed heap blocks? Not sure about that.)
* We could do something like the size of all heap allocations plus other notable non-heap allocations (like code pages allocated by the JITs). This requires some internal accounting and so is hard to compare against other browsers.
* What do we do about multiple processes? Measure them all and sum the numbers?
* There seems to be a lot of non-determinism in browser memory usage, which makes pinpointing small regressions/improvements difficult.
* Do we only measure peak usage? Or do we want to open and close lots of tabs to try to catch leaks?
* jseward's [http://blog.mozilla.com/jseward/2011/01/27/profiling-the-browsers-virtual-memory-behaviour/ VM-sim extension to Cachegrind] takes a different tack; it measures page faults on a quasi-idealized machine. It's slow, though.
== A Clear Goal ==
Some ideas:
* A certain percentage better than current, for whatever metric we choose. Arbitrary, but clear.
* Within a certain percentage of other browsers (possibly including Firefox 3.6). Requires a metric that can be measured in other browsers easily.
* No obvious leaks. Could be aimed for if we measure after closing lots of tabs.
* Keep going until we can open N copies of TechCrunch on some standard phone? (I.e. until the mobile guys stop complaining)
* Keep going until commenters on slashdot/Ars/reddit/etc stop complaining??
== Better Profiling ==
nnethercote has plans for greatly improving about:memory. See [http://blog.mozilla.com/nnethercote/2011/02/09/a-vision-for-better-memory-profiling-with-aboutmemory/ this blog post] for details. These will make it much easier for everyone to understand how Firefox is using memory.
== A Straw-man Proposal ==
Just to get discussion going...
=== Benchmark ===
Something like Membuster: open a bunch of popular and/or interesting (eg. memory intensive, ones that caused problems in the past) pages, one per tab.
Not too many though; I suspect that opening 100 won't give us much more information that 10 would, and it should be runnable by devs easily.
Use recorded versions for consistent results, if possible, and make those recorded versions easy to use by devs.
=== Metrics ===
* Peak RSS: easy to measure (on all platforms -- I think?) across all browsers, has some reflection on actual performance.
* Peak total: easy to measure across all browsers, interesting for the address space exhaustion issue.
So, only two metrics per browser per run. Leaks aren't addressed; might be able to included them if we measure at startup and shutdown (after all tabs have closed).
=== Goal ===
Reduce peak RSS by 1/3.
(Who knows? Probably need to measure other browsers to get a sense of what the goal should be.)
== Pushing This Forward ==
nnethercote and jmuizelaar have volunteered to co-lead the MemShrink effort.
MemShrink could be modelled on CrashKill and CritSmash. One problem with that is that neither of them are in Mountain View -- jmuizelaar is in the Toronto office (not too bad), nnethercote works at home in a PST+17/+19 timezone. So having CrashKill/CritSmash-style meetings will be difficult.
The MemShrink effort can also include leaks and quasi-leaks in its jurisdiction, even if areweslimyet.com doesn't demonstrate them.
