User:Rdoherty/Velocity notes

Slides

Jiffy

• Can measure 0-100% of all page loads
• Measure anything
• Measure when page is useable
• Real/near time reporting
• No impact on page perf. (very little impact)
• Components
  • jiffy.js
  • apache config
  • Perl & php
  • database (oracle right now)
• Mark & measure
• Immediate or batch submits
• Default browser even measurements
• Firebug plugin

Measuring Performance

• Tools:
  • Keynote KITE
  • Coradiant Truesight + Web.I, Tealeaf, Jiffy
  • Internal
    • HP/Mercury SiteScope + BAC, Nagios,
  • External synthetic monitoring
    • keynote, gomez, webmetrics
  • Network analysis
    • wireshark, clearsight, ACE, NetQoS
  • In depth
    • Opnet Panorama, Introscope, Quest (DB),
  • Load testing
    • LoadRunner, Jmeter, ab
• Many drawbacks to testing tools. Some cause extra load, crash servers, not accurate.
• Hard to duplicate production systems
• Turning star rating from 50 images to 2 on Netflix actually degraded performance. Don't attach event handlers to everything, use containers.

Hotmail's Performance Tuning practices

• Performance should be core from the beginning
• Customers don't care what performance is
• Best performance gains are simplest of all
• Laws of physics
  • Use less bytes
  • HTML is faster (don't build UI with javascript)
  • Asymmetric bandwidths can bottleneck upstream
• Max bandwidth is reliant on latency. 300ms latency == 300kb/s
  • bring content closer to user
  • minimize page weights
• Utilize closed feedback system
• Trim page weights
  • Render more content on server side
  • Trim image sizes
  • Use cache control, expiration
  • Optimize between inline & stand alone css & js
• Trim page weights (upstream)
  • Trim cookie sizes
  • Trim down # of redirects
  • Don't set cookies at root path or domain
• Bandwidth efficiency
  • multiple domains
  • JIT downloading
  • Control sequencing of downloading

Live Search

• New redesign with lots more assets
• Netn
• Inline css & js

High performance AJAX

slideshare.net/julien.lecomte/high-performance-ajax-applications

• Plan for performance from day 1
• Don't do anything until necessary
• Cheat: show and update before it's completed
• Keep UI responsive at all times
• Test performance using a setup similar to users' environment
• Profile code during development
• Automate profiling & performance testing
• Keep historical data
• Reduce total uncompressed code
• Close all tags
• Flush apache buffer early
• Don't wait for onload
• Post-load scripts
• Pre-load next page's assets
• Javascript
  • reduce the amount of symbolic look-up (use var inside {}'s)
  • stay away from with keyword
  • cache results of expensive lookups
  • Reduce prototype chain lookups
  • Bind properties/functions to objects prototype constructor
  • Don't use eval
  • Pass anonymous functions to setTimeout and setInterval,not strings
  • Optimize string concatenation
  • Don't user regexp constructor
  • chain long javascript processes using setTimeout
  • primitive operations are better than function calls
  • avoid for...in in performance critical sections
  • Use innerhtml
  • Use clonenode
  • limit # of event handlers. use event delegation
  • Limit reflows of document. Happens whenever anything happens in the dom
  • Batch style changes
  • Modify elements with display:none
  • downshift your code: throttle frequent and expensive actions
• CSS
  • use sprites
  • don't use js for layout
  • avoid expressions
  • avoid IE filters
• AJAX
  • don't user synchronous xhr
  • handle timeouts
  • update UI when request is sent
  • lock smallest possible parts of ui
  • use json
  • push, don't poll

Cadillac or Nascar?

• Evaluating 1 app written in PHP, JavaEE and RoR. Social Event calendar
• Use MySQL 5.1
• Apache
  • do not load modules you don't use
• PHP
  • increase realpath_cache_size if you have a lot of files
• Memcache
  • use 4 processes
  • don't run on server that is also doing web stuff
• Mysql
  • tune queries
  • joins over subqueries
  • use limits
  • separate read/write dbs

Fiddler

• Can simulate modem speeds

AOL page tester

• webpagetest.org

Even faster websites

• Very little time is spent retrieving html
• New rules
  • split initial payload
  • load scripts without blocking
  • don't scatter inline scripts
• Doloto (MS testing utility, not open or available :( )
• Parallel script loading
  • XHR eval. Fetch js as xmlhttprequest, eval it.
  • Must be on same domain as main page
  • Must refactor script
  • XHR injection
  • appending javascript from xhr request to doc head
  • script in iframe
  • script must target parent page
  • unknown performance penalty
  • Script dom element
  • create script element, set source, append to head of document
  • no need to refactor
  • Script defer
  • <script defer
  • only works in IE
  • no refactoring
  • Document.write script tag
  • only works in IE
  • only adds parallelization for scripts
  • all document.writes must be in same script block
• Browser busy indicators
  • Necessary if scripts will perform an action the user is waiting for
• Ensure/avoid ordered execution
• Inline scripts block as well
  • Use setTimeout to remove blockig behaviour (>250ms for FF)
• Firefox blocks parallel downloads when downloading stylesheets
  • IE doesn't, except when stylesheet is followed by inline script
  • Move inline scripts above everything

Performance Plumbing

• ANYCAST
  • net address and routing scheme used for proximity
  • also useful for load absorption

Capacity Management for Web Operations

• Book coming out soon.
• Capacity != Performance
• Forget about performance
• Measure what you have right now
• Automate stuff
  • system imager
  • ganglia
• Good measurement tools
  • record and store
  • metrics in/out
  • custom metrics
  • easily compare
  • lightweight
• Ganglia!
• Ceilings
  • most amount of work a box will do before failing/degrading
• Forget benchmarking
  • use production data
  • use live production loads (divert % of traffic to new features/systems, etc)
  • pull boxes out of rotation, see where others fall over
• Find your 'safe' ceiling, what you are comfortable with
• Forecasting
  • Example with webservers
  • Graph your utilization, find trends, extrapolate
  • Calculate your ceiling based on # of machines you have how much traffic they can handle
  • Automate your forecasting
  • Calculate 'days remaining' until you have to add servers
  • fityk.sf.net * graphing tool
  • Storage consumption
• Create high & low water marks
  • alerts if you go below or above limits
• Create dashboard that shows your % utilization and days left on your current architecture
• Scale diagonally
  • add more horsepower per box
  • you can use less machines
  • less power
  • less racks
• 3.4TB/day consumed on Flickr
• 2nd order effects
  • Add memcache, decreases DB load, increases web server load
  • Add web servers, increases DB load
• Puppet
• DSH
  • distributed shell
• Disable heavy features via configuration flags
  • Better to have something up than nothing
• Host your outage blog/status/page in a different datacenter & domain
• Tell your user what's going on
• Add ability to turn dynamic content into static content
• Some Y! properties have ability to turn entire site into static content (big red button)
• Developers need to think like sys admins and have access to same data & configuration
• Very verbose logging. who, what, when for pushes correlated with ganglia graphs
• Small pushes are better than large pushes every 2 weeks

Squid and Varnish

• Squid sucks, Varnish is hella fast (more than 10x)

This is your site on Ads

• Nearly all ad networks use document.write, can't be loaded after page has loaded
• Ads don't gzip or minify. They load too much crap (beacons, js, image, etc)