Breakpad/Status Meetings/2017-02-01

« previous meeting — index – next week » ^create?

Meeting Info

Breakpad status meetings occur on Wed at 10:00am Pacific Time.

Conference numbers:

   Vidyo: Stability 
   650-903-0800 x92 conf 98200#
   800-707-2533 (pin 369) conf 98200# 

IRC backchannel: #breakpad
Mountain View: Dancing Baby (3rd floor)

Operations Updates

• miles killed a bunch of random instances
  • ~$2100 / month in savings

• stage submitter problems
  • on wednesday, january 25th, it stopped submitting
    • turned out it was looking at the rabbitmq cluster that Miles took out (it should have been unused)
    • we updated the stage submitter and -prod crashmover configuration to use cloudamqp cluster
  • on tuesday, january 31st, it kicked up disk space alert
    • bunch of orphaned processes again
    • Will tweaked the run_submitter.sh script adding the -once arg for envconsul
    • increased crashes to prod -> increased crashes to stage submitter -> more logging -> more disk used by log files

• crash collection rate in prod increased 3x
  • door hanger to submit old crashes was turned on again
  • this will happen for the first ~5 betas of every cycle
  • see: https://bugzilla.mozilla.org/show_bug.cgi?id=1303067

• access to leeroy instructions are on socorro-dev
  • needs duo push, so it needs the duo app

• we've been dropping old es indices faster
  • due to higher load
  • at this point we can manually shorten the retention period
  • shard count is higher now, could we scale up?
  • let's reduce retention for now

Project Updates

Deployment Triage

• https://whatsdeployed.io/s-7M7

PR Triage

• http://prs.mozilla.io/mozilla/socorro

Major Projects

Splitting out collector (Antenna)

• (willkg) checked whether data saved by Antenna is the same as the Socorro collector; fixed a couple of issues--should be good to go now
• (willkg, miles) checked -stage environment and dashboards--all set for a load test on -stage
• (willkg, miles, lonnen) worked out two viable ways to get crash siphoning from -prod to -stage working with Antenna; pushing off any work until we need it with the hopes that AWS Lambda supports Python 3 by then
• (mbrandt) load tests in progress; they changed the framework to asyncio, so we need to rework to that

Deprecation rampage

• no update

Processor rewrite

• [this update intentionally left blank]

Upgrading elasticsearch

• migration plan: https://public.etherpad-mozilla.org/p/socorro-es5-migration-plan
• (adrian) Use this opportunity to start using aliases for all our indices?

Other Business

• (mbrandt) Geckodriver
  • webdriver will be deprecated sometime (maybe 6 months out)
  • mbrandt is trying to figure out when we are forced to move to keep up to date
  • mbrandt to follow up with devs working out the problems locally
  • mbrandt to wordsmith the readme

Travel, etc

• peterbe afk through Feb

Links