Breakpad/Status Meetings/2017-02-01

From MozillaWiki
Jump to: navigation, search

« previous meetingindexnext week » create?

Meeting Info

Breakpad status meetings occur on Wed at 10:00am Pacific Time.

Conference numbers:

   Vidyo: Stability 
   650-903-0800 x92 conf 98200#
   800-707-2533 (pin 369) conf 98200# 

IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)

Operations Updates

  • miles killed a bunch of random instances
    • ~$2100 / month in savings
  • stage submitter problems
    • on wednesday, january 25th, it stopped submitting
      • turned out it was looking at the rabbitmq cluster that Miles took out (it should have been unused)
      • we updated the stage submitter and -prod crashmover configuration to use cloudamqp cluster
    • on tuesday, january 31st, it kicked up disk space alert
      • bunch of orphaned processes again
      • Will tweaked the run_submitter.sh script adding the -once arg for envconsul
      • increased crashes to prod -> increased crashes to stage submitter -> more logging -> more disk used by log files
  • access to leeroy instructions are on socorro-dev
    • needs duo push, so it needs the duo app
  • we've been dropping old es indices faster
    • due to higher load
    • at this point we can manually shorten the retention period
    • shard count is higher now, could we scale up?
    • let's reduce retention for now

Project Updates

Deployment Triage

PR Triage


Major Projects

Splitting out collector (Antenna)

  • (willkg) checked whether data saved by Antenna is the same as the Socorro collector; fixed a couple of issues--should be good to go now
  • (willkg, miles) checked -stage environment and dashboards--all set for a load test on -stage
  • (willkg, miles, lonnen) worked out two viable ways to get crash siphoning from -prod to -stage working with Antenna; pushing off any work until we need it with the hopes that AWS Lambda supports Python 3 by then
  • (mbrandt) load tests in progress; they changed the framework to asyncio, so we need to rework to that

Deprecation rampage

  • no update

Processor rewrite

  • [this update intentionally left blank]

Upgrading elasticsearch

Other Business

  • (mbrandt) Geckodriver
    • webdriver will be deprecated sometime (maybe 6 months out)
    • mbrandt is trying to figure out when we are forced to move to keep up to date
    • mbrandt to follow up with devs working out the problems locally
    • mbrandt to wordsmith the readme

Travel, etc

  • peterbe afk through Feb

Links